Padding input data for artificial intelligence accelerators

Abstract

The processing of input data for transmission to a data consumer, such as an artificial intelligence engine, is performed by arranging the input data into a uniform structure, which consists of sticks of data combined to form a page of sticks. A stick is any appropriately sized set of input data elements, according to which the size of the stick is fixed. A masking pattern is established for sticks of data with certain ranges of invalid data for consumption of partial sticks, while maintaining the validity of the input data being forwarded. The mask pattern is derived based on the Active Mask and Set Value (SAMV) instruction. The derived mask pattern is carried over to subsequent load instructions to the data consumer. (Figure 5)

Description

[Technical field]

[0001] The present invention relates generally to the field of hardware accelerators, and more particularly to accelerating artificial intelligence (AI) and machine learning applications. [Background technology]

[0002] AI accelerators are a class of specialized hardware accelerators designed to process AI and machine learning applications with high-performance parallel computers that are purpose-built to efficiently process AI workloads such as artificial neural networks and machine learning. Typical applications include algorithms for robotics, the Internet of Things, and other data-intensive or sensor-driven tasks.

[0003] Artificial intelligence accelerators are typically co-located with the processor and memory subsystem in a computing system: they may be embedded on the same chip, attached to the processor system by external I / O interfaces, or attached remotely in a cloud computing environment. Summary of the Invention

[0004] In one aspect of the invention, a method, computer program product, and system for padding data while loading data from a memory to a data consumer includes receiving a load instruction including padding instructions, the load instruction including a load instruction to read input data arranged in a stick-like layout from a memory to the data consumer, the padding instructions including replacement values ​​and padding parameters for elements to be masked, deriving a mask pattern from the parameters specified in the padding instructions, padding the input data by masking invalid elements of the stick-like layout, and generating a set of data sticks including padded data for the data consumer. Applying padding on-the-fly from the load instruction via the padding instructions has a significant performance benefit by eliminating any pre-processing steps. The parameters specified in the padding instructions include layout parameters and pad range parameters. The stick layout is separable into a predefined number of slices, and masking the invalid elements includes identifying invalid slices in the stick, where the invalid slices contain invalid elements, and determining a starting slice having a portion of valid elements and a portion of invalid elements. The padding instruction provides a data format specified in a 6-bit instruction that includes a data format, a cross slice dimension type, and a leading dimension of the stick.

[0005] In another aspect of the present invention, a method, computer program product, and system for padding data while loading data from a memory to a data consumer includes transmitting a set of data sticks to the data consumer in response to a load command.

[0006] In yet another aspect of the present invention, a method, computer program product, and system for padding data while loading the data from a memory to a data consumer includes decoding a padding instruction to determine a replacement value and a mask pattern.

[0007] In yet another aspect of the invention, a method, computer program product, and system includes: decoding a padding instruction to determine a replacement pad value and a mask pattern padding parameters for a set of sticked input data, the padding instruction being embedded in a first load instruction of the sticked input data; deriving a mask pattern from the padding parameters; generating a set of data sticks from the set of sticked input data for a requesting consumer, the set of data sticks padded according to the derived mask pattern including the replacement pad value; and applying the derived mask pattern and the associated replacement pad value to a subsequent load instruction to load the subsequent input data, the subsequent load instruction being equivalent to the first load instruction, the subsequent input data being padded during transmission to the requesting consumer. The set of sticked input data is arranged as sticks, each stick separable into a predefined number of slices. The replacement pad value is determined by reference to a pad value register file.

[0008] In yet another aspect of the invention, a method, computer program product, and system includes padding a set of stickified input data by masking invalid elements of each stick to generate a set of data sticks, where masking the invalid elements of the sticks includes identifying invalid slices in the sticks, where the invalid slices include invalid elements, and determining a starting slice having a portion of valid elements and a portion of invalid elements. The set of stickified input data is arranged as sticks, where each stick is separable into a predefined number of slices. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram of a first embodiment of a system according to the invention; [Diagram 2] 4 is a flow chart illustrating a first method performed at least in part by the system of the first embodiment. [Diagram 3] FIG. 2 is a schematic diagram of the machine logic (eg, software) portion of the system of the first embodiment. [Figure 4] 4D tensor diagrams illustrating information generated by and / or useful in understanding embodiments of the present invention. [Diagram 5] 1 is a hierarchical diagram of 4D tensors illustrating information that is generated by and / or useful in understanding embodiments of the present invention. [Figure 6] FIG. 2 is a schematic diagram of a second embodiment of the system according to the invention. [Figure 7] 6 is a flow chart illustrating a second method performed at least in part by the system of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The processing of input data for transmission to a data consumer, such as an artificial intelligence engine, is performed by arranging the input data into a uniform structure, which consists of sticks of data combined to form pages of sticks. A masking pattern is established for sticks of data that have a certain range of invalid data for consumption of partial sticks while maintaining the validity of the input data being forwarded. The mask pattern is derived based on a set-active-mask-and-value (SAMV) instruction. The derived mask pattern is carried over to subsequent load instructions to the data consumer.

[0011] The present invention may be a system, method, and / or computer program product, which may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0012] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0013] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0014] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and traditional procedural programming languages ​​such as the “C” programming language, or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0015] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0016] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions for performing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0017] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operable steps to create a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0018] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as one step, or may be executed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0019] The present invention will now be described in detail with reference to the figures. Figure 1 is a functional block diagram illustrating various parts of a networked computer system 100 according to an embodiment of the present invention, including an accelerator subsystem 102, a neural network subsystem 104, an artificial intelligence engine 105, data consumer subsystems 106, 108, workload subsystems 110, 112, an image record 111, a text record 113, a communication network 114, a mask computer 200, a communication unit 202, a processor set 204, an input / output (I / O) interface set 206, a memory device 208, a persistent storage device 210, a display device 212, an external device set 214, a random access memory (RAM) device 230, a cache memory device 232, a padding engine 240, a data mover program 300, and a padding register 302.

[0020] Subsystem 102 is in many respects representative of various computer subsystems of the present invention, and therefore, several portions of subsystem 102 will now be described in the following paragraphs.

[0021] Subsystem 102 may be a laptop computer, tablet computer, netbook computer, personal computer (PC), desktop computer, personal digital assistant (PDA), smart phone, or any programmable electronic device capable of communicating with a client subsystem over network 114. Program 300 is a collection of machine-readable instructions and / or data used to create, manage, and control certain software functions described below.

[0022] Subsystem 102 may communicate with other computer subsystems via network 114. Network 114 may be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and may include wired, wireless, or fiber optic communications. In general, network 114 may be any combination of connections and protocols that support communications between server and client subsystems.

[0023] Subsystem 102 is shown as a block diagram with a number of double-headed arrows. These double-headed arrows (which do not have separate reference numbers) represent a communications fabric that provides communication between the various components of subsystem 102. This communications fabric may be implemented using any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, the communications fabric may be implemented, at least in part, using one or more buses.

[0024] Memory 208 and persistent storage 210 are computer-readable storage media. In general, memory 208 may include any suitable volatile or non-volatile computer-readable storage media. Additionally, it should be noted that, currently or in the foreseeable future, or both, (i) external device 214 may be able to provide some or all of the memory for subsystem 102, or (ii) devices external to subsystem 102 may be able to provide memory for subsystem 102.

[0025] The program 300 is stored in persistent storage 210 for access and / or execution by one or more of the respective computer processors 204, typically through one or more memories of memory 208. Persistent storage 210 (i) is at least more persistent than a transient signal, (ii) stores the program (including its soft logic and / or data) in a tangible medium (such as a magnetic or optical field), and (iii) is substantially less persistent than the permanent storage. Alternatively, data storage may be more persistent and / or permanent than the type of storage provided by persistent storage 210.

[0026] The program 300 may include machine-readable and executable instructions and / or tangible data (i.e., the type of data stored in a database). In this particular embodiment, persistent storage 210 includes a magnetic hard disk drive. Among other possible variations, persistent storage 210 may include a solid-state hard drive, a semiconductor memory device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0027] The media used by persistent storage 210 may also be removable. For example, a removable hard drive may be used for persistent storage 210. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 210.

[0028] The communications unit 202, in these examples, provides for communication with other data processing systems or devices external to the subsystem 102. In these examples, the communications unit 202 includes one or more network interface cards. The communications unit 202 may provide for communication by using either or both physical and wireless communications links. Any software modules described herein may be downloaded to persistent storage (such as persistent storage 210) through a communications unit (such as the communications unit 202).

[0029] The I / O interface set 206 allows input and output of data to and from other devices that may be locally connected in data communication with the computer 200. For example, the I / O interface set 206 provides a connection to an external device set 214. The external device set 214 typically includes devices such as a keyboard, a keypad, a touch screen, or any other suitable input device, or a combination thereof. The external device set 214 may also include portable computer-readable storage media, such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the invention, such as the program 300, may be stored on such portable computer-readable storage media. In these embodiments, the relevant software may (or may not) be loaded into the persistent storage 210 in its entirety or such software may be partially loaded into the persistent storage 210 via the I / O interface set 206. The I / O interface set 206 also connects in data communication with a display device 212.

[0030] Display device 212 provides a mechanism for displaying data to a user and may be, for example, a computer monitor or a smart phone display screen.

[0031] The programs described herein are identified based on the application for which they are implemented in a particular embodiment of the invention, however, it will be understood that the nomenclature of specific programs herein is used merely for convenience, and thus the invention should not be limited to use only with the particular application identified and / or implied by such nomenclature.

[0032] The data mover program 300 operates to process input data for transmission to a data consumer, such as an artificial intelligence engine. The input data is arranged in a uniform structure for easy consumption, which consists of sticks that combine to form pages. The data mover program 300 establishes a masking pattern for sticks of data that have certain ranges of data that are invalid. The padding process achieves consumption of partial sticks while maintaining the validity of the input data being transferred. The data mover program derives a mask pattern for the input data during transfer to the data consumer with a specified load instruction or loading program that may include a SAMV instruction to set the active mask and pad values. The derived mask pattern is carried over to subsequent load instructions to the data consumer.

[0033] Some embodiments of the present invention recognize the following facts, potential problems, and / or potential areas for improvement with respect to the current state of the art.

[0034] An example of standard length input data is that found in a high bandwidth memory (HBM), which can only be addressed at a granularity of 128 bytes. Thus, a sequence of 128 bytes of input data is referred to as a "stick" as the term is used herein. In defining the data layout of each data structure constructed, certain dimensions constitute a stick. Partitioning a tensor into pages of sticks of data elements is referred to herein as "stickification," or simply arranging a tensor into sticks of data elements. As used herein, the term "stick" can be defined as any appropriately sized set of input data elements, according to which the size of the stick is fixed. According to some embodiments of the present invention, the size of the stick is the SIMD processing power of the accelerator.

[0035] If the workload parameter (N) in a given dimension is not an exact multiple of the stick size (S) in that dimension, then a portion of the last stick needs to be padded to fill empty, i.e., invalid elements created by input data less than 128 bytes. Here, the last stick of input data is of interest because the other sticks in the array of input data are complete sets of valid input data. Some embodiments of the present invention are directed to fast padding of invalid elements in AI accelerators. As previously described, the invalid elements of each input data can be found in the last stick of the stick that contains that input data.

[0036] For example, for numeric format FP16, the stick dimension of the input activation can be chosen to be the channel dimension "in". For a stick size of 128 bytes, the stick channel dimension will be 64 elements, each element being 16 bits, for a total of 128 bytes for each stick. If the workload parameter N is not a multiple of the stick size of 128 bytes, dividing the workload parameter by 128 will produce a remainder. The remainder indicates how many elements of the last stick should be padded, since they would be invalid if not padded. N in %S in For a given scenario where N in %S in ) = PE, where N in is the workload value, and S in is the stick size and PE is the number of padded elements of the final stick that composes the workload.

[0037] Some embodiments of the present invention are directed to an instruction format that specifies a mask pattern and selects a pad value from a register file to replace the masked elements in the stick, leaving the unmasked elements unchanged. One example of a suitable instruction format is referred to herein as the SAMV (Set Active Mask and Value) instruction. The pad value itself may vary depending on application-specific requirements, but in many cases a pad value of zero will be preferred.

[0038] Figure 2 shows a flow chart 250 illustrating a first method according to the invention. Figure 3 shows a program 300 for performing at least some of the method steps of flow chart 250. The method and associated software will now be described over the following paragraphs with broad reference to Figure 2 (for method step blocks) and Figure 3 (for software blocks).

[0039] Processing begins at step S255, where a stick module ("mod") 355 generates input data as a set of stickified input data into sticks of data. As described herein, the term "stick" refers to a uniform portion of input data organized for accelerated consumption by a target application. As described in more detail below, an exemplary stick size is 128 bytes, or 64 elements within each stick. Arranging input data into sticks of data is referred to herein as "stickification." A specific process of stickification of input data is provided herein. When forming a stick of data, it is common for the input data to not fit exactly into a 128-byte portion, but some unused bits remain within one stick for a given data set.

[0040] Processing proceeds to step S260, where the load mod 360 receives a load instruction or program for loading input data from memory to the data consumer. In this example, input data including images from the image record store 111 is loaded from the workload subsystem 110 (FIG. 1). Further, in this example, the load instruction directs transmission of the input data to the AI ​​engine 105 of the neural network subsystem 104. The load instruction may include (i) an instruction to read data from memory to the consumer, (ii) an instruction to copy intermediate values ​​to a register file, (iii) a “set active mask and value” instruction (SAMV instruction) for padding, and (iv) an instruction to loop, i.e., repeat, a sequence of instructions for a given number of cycles.

[0041] Processing proceeds to step S265 where the input modifier 365 identifies the input data and padding values ​​to be stuck. Upon receiving the load command, the data mover program 300 proceeds to locate the input data and assign padding values. The input data location may be stored at a remote location or on the computer system executing the padding program. For purposes of this disclosure, the input data is assumed to be consumed by a target application, such as a neural network AI engine or machine learning application. The padding value is associated with that application and provides masking of invalid elements of the required sequence length when the input data does not meet the length. In this example, the padding value is recorded in the padding register 302 for use in padding the identified input data. The stored padding value is later applied to the elements to be masked.

[0042] Processing proceeds to step S270 where the decode mod 370 decodes the SAMV instruction. The decoded SAMV instruction initiates reading padding values ​​from registers and storing the padding values ​​for use. The SAMV instruction also performs mask pattern operations and generates an active mask for the input data read from memory.

[0043] Processing proceeds to step S275 where pattern modifier 375 derives a mask pattern for the stickified input data. The mask pattern is derived from layout and pad range parameters encoded within bit fields of the SAMV instruction. The SAMV instruction provides controlled displacement of pad elements along the inner and outer dimensions within the stick, which may be divided into slices. A detailed description of the mask pattern derivation is provided below.

[0044] Processing proceeds to step S280 where the mask mod 380 generates a mask based on the mask pattern to pad the input data. The mask mod 380 generates the mask based on parameters derived from the SAMV instruction. In this example, assume that image records from the workload 110 are sent to the AI ​​engine 105. The image records are arranged in a stick-like format and divided into slices for identification and padding of invalid elements. The mask mod establishes a set of padded sticks of data in the form of pages of input data for accelerated consumption by the target consumer.

[0045] Processing continues to step S285 where a loop mod 385 applies the mask pattern to subsequent input data. In this example, a loop instruction is received in step S255 where the loop instruction indicates how many data sets should be processed according to SAMV instructions for padding invalid elements of the input data arranged in a uniform data structure. In this way, a single SAMV instruction is applied to multiple cycles of mask generation as well as the first input data set.

[0046] Further embodiments of the invention will be described in accordance with and in the paragraphs that refer to Figures 4 to 7.

[0047] Some embodiments of the invention are directed to implementing a load unit for padding input data while loading a stick of data from a memory or scratch pad store. Some embodiments of the invention are directed to supporting various element sizes and hierarchical stick-like layouts.

[0048] According to some embodiments of the present invention, values ​​for replacing pad elements are stored in a register file of a load unit during initialization or during run-time.

[0049] Some embodiments of the present invention are directed to deriving a mask pattern from layout and pad range parameters encoded within bit fields of an instruction, such as the SAMV instruction. The act of deriving a mask pattern provides for real-time generation of the mask pattern during transfer to a consuming device. Cross-slice and intra-slice masking operates to reduce mask decode complexity through the use of coarse-grained (cross-slice) and fine-grained (intra-slice) masks. Additionally, the compact nature of the 6-bit stick data layout scheme ensures a small instruction footprint. Hierarchical layouts having both inner and outer dimensions within a slice are supported by the SAMV instruction such that replacement of pad elements can be controlled to occur along the inner and outer dimensions within the slice.

[0050] Some embodiments of the invention are directed to a 6-bit instruction format that uses 2 bits to indicate the element size, 3 bits to indicate the length of the innermost dimension, and 1 bit to indicate whether the innermost dimension is of cross-slice or intra-slice type.

[0051] According to some embodiments of the present invention, when a SAMV instruction is executed, the derived mask and replacement values ​​are saved and applied by all subsequent load instructions to pad the stick accordingly. Doing so allows some embodiments of the present invention to perform in-stick padding on the fly without any software overhead other than the "SAMV instruction" to set the padding rules for all subsequent loads. Furthermore, because the pad area is defined in terms of elements rather than bytes, the precision of the workload can be changed without changing the definition of the pad area.

[0052] Figure 4 illustrates a 4D tensor 400 that represents input data elements with certain characteristics or dimensions, including height H, width W, channels C, and batch size N. In this example, the values ​​shown in Table 1 are assigned to these dimensions.

[0053] [Table 1]

[0054] The original tensor described in Table 1 is reconstructed into a number of pages per dimension according to the hierarchical partitioning of data elements. Each page consists of a memory structure referred to herein as a "stick." In this example, each stick is quantized to 128 bytes and each page is quantized to 32 sticks. Thus, most pages are full pages 532 in that they are fully utilized with valid elements. A stick may correspond to the width of the data bus and / or the effective SIMD (single instruction multiple data) width. For example, 64 elements x 2 bytes / element = 128 bytes.

[0055] Because the number contains fractional parts, the new page structure may contain partial pages 533 at its ends that are not fully utilized. In that case, only the utilized sticks should be loaded. This creates address gaps 534 at the ends that need to be compensated for in order to stream data.

[0056] Figure 5 shows a hierarchical partitioning of the data elements of Figure 4 into data components including pages 531, 532, 533 and sticks 536 within each page. The component dimensions QH, QW, QC, and QN are derived from the 4D tensor dimensions.

[0057] The calculation of the component dimensions is shown in Table 2.

[0058] [Table 2]

[0059] The new data structure resulting in hierarchical partitioning may contain empty elements at the ends if the original tensor size in the stick dimension does not divide evenly into the 128-byte stick. In this example where the dimension of interest is the channel C, the value of C must divide evenly into 128 bytes to avoid having empty elements. If the example 4D tensor had 128 elements that are 2 bytes each (C=128), there can be no empty elements in the hierarchical partitioning of the 4D tensor. Any data present at the addresses of elements that appear empty is invalid and should not contribute to the result. Since SIMD operations are performed on the entire stick, any invalid regions need to be padded with a suitable value, such as "0" from a register, so that they do not change the result value or cause exception 531. Referring back to this example, the value of C is 120 elements, so there are 16-byte empty elements, or 8 elements that are empty and need to be padded before processing.

[0060] As shown in Figure 5, many pages are full pages 532 that are fully utilized with valid elements. Other pages are partial sticks 531 or partial pages 533 with address gaps 534 at the ends. As described herein, pages with partial sticks 531 are dynamically padded according to an algorithm performed during a mask operation before sending the data volume to the consumer.

[0061] Note that the order of dimensions of a given input tensor may be changed to match the processing order of the accelerator. For example, the original tensor is shown in Table 1 in NHWC order. If we refer to the page and reconstruct the tensor, the order shown in Table 2 is NCHW, which is changed from the order in Table 1.

[0062] FIG. 6 illustrates a system for on-the-fly padding of data paths to consumer logic for single instruction multiple data (SIMD) parallel processing or for use by an arithmetic logic unit (ALU). The term on-the-fly padding refers to performing a padding, i.e., masking, operation while transferring input data in a data volume suitable for the data consumer. The system of FIG. 6 includes a load unit subsystem 602, an instruction fetch module 610, an instruction buffer 612, an instruction decode module 620, a register file module 630, a pad value register 640, a mask decode module 650, a mask register 660, a memory subsystem 604, a mask operation subsystem 606, and a consumer subsystem 608.

[0063] Some embodiments of the present invention are directed to a load unit subsystem including instruction fetch logic 610 and instruction buffer 612 for executing program related memory load operations. Alternatively, the instruction fetch logic may be located outside the load unit subsystem and shared with other load unit subsystems (not shown). Decode logic 620 decodes the fetched instructions and generates control signals to register file 630, mask decode 650, and actual memory blocks 604.

[0064] A set of instructions for performing some embodiments of the present invention may include (i) load (to read data from memory to a consumer), (ii) regcopy (to copy an immediate value to register file 630), (iii) SAMV (set active mask and value) instructions (to set the active mask and value for padding), and (iv) loop (to repeat a sequence of instructions a given number of times).

[0065] As mentioned above, the SAMV instruction is read as part of the instruction for some embodiments of the present invention. The SAMV instruction specifies a mask pattern and selects a pad value from the register file 630 to replace the masked elements in the stick and leaves the non-masked elements unchanged. Additionally, the pad value in the register file is pre-stored or pre-defined and held in the pad value register 640. The SAMV instruction parameterizes the mask decode logic to generate an active mask. The active mask is used by the mask operation logic 606 to apply the selected pad value to the masked elements of the subsequent data read from the memory 604. According to some embodiments of the present invention, the instruction decode module and the mask decode module are combined into a single module for decoding. Additionally, the memory module 604 may reside in the load unit 602.

[0066] A mask operation 606 is applied to all memory reads on the data path to a consumer module 608. The mask determines if any elements are replaced by a pad value, and which of the identified elements should be replaced.

[0067] Figure 7 shows a flowchart 700 illustrating a second method according to one embodiment of the present invention. Figure 6 shows a system 600 for performing at least some of the method steps of flowchart 700. The method and associated software will now be described over the following paragraphs with broad reference to Figure 7 (for method step blocks) and Figure 8 (for system blocks).

[0068] Processing begins at step S702, where the data organization within a stick is derived from the 6-bit layout field of the SAMV instruction. The stick organization defined by the SAMV instruction includes slice-specific data. Each stick of 128 bytes can be further divided into 16-byte slices, with each slice consisting of two dimensions, A and B. There are eight slices, X, X, and X. xsl = 8 forms a stick, and the slices are in a single dimension [L A ] or [L B ]. Dimensions that exist across multiple slices are cross-slice dimensions, denoted as "xsl", and dimensions that exist only within one slice are intra-slice dimensions, denoted as "wsl". Thus, the data layout in a stick has the following format: {[X xsl ][L wsl ][L xsl ]}or, {[X xsl ][L xsl ][L wsl ]} Limited to. X xsl Note that L is always equal to 8, whether the data or number format is FP16, FP8, INT4, or INT2. wsl *L xsl =16 bytes / element length (element length is expressed in bytes / element), which means 8 elements for FP16, 16 elements for FP8, 32 elements for INT4, or 64 elements for INT2.

[0069] Only 6 bits are used to specify the data layout in the stick: 2 bits indicate the data format and 1 bit indicates whether the cross slice dimension is innermost, i.e., [L wsl ][L xsl ] or [L xsl ][L wsl], and 3 bits indicate either the cross-slice dimension length or the intra-slice dimension length, where [L xsl ] and [L wsl Note that the length of the slice dimension only uses 3 bits because [ ] is a power of 2 with the choices being {64, 32, 16, 8, 4, 2, 1}. Thus, the input 7 is invalid in the set {6, 5, 4, 3, 2, 1, 0}, which represents the set {64, 32, 16, 8, 4, 2, 1}.

[0070] The hierarchical division of data elements shown in Figure 5 is referred to in developing the following mask generation algorithm. The required values ​​are provided as input values ​​by the SAMV instruction for padding. X xsl =sticksz / 128=8, elsz=2 1-layout[5:4] , inner_dim=(layout[3]==0)?xsl:wsl L wsl =2 layout[2:0] , L xsl =128 / (elsz*L wsl ) where sticksz is the specified stick size, such as 128 bytes, and layout refers to the 6-bit layout field of the SAMV instruction (see Table 3 below).

[0071] Parameter SliceID xsl and Valid xsl is derived from the layout parameters and the number of valid elements n in the stick. The layout parameters and the number of valid elements are provided in the SAMV instruction. The derivation is as follows:

number

number

[0072] Processing continues at step S704 where a slice count "s" is set to zero. The process for padding invalid elements of a stick begins by "slicing" the selected stick into a set of slices, such as 8 slices in this example. Processing begins with slice 0 of the 8-slice stick and ends with slice 7. As described herein, if the input data is not evenly divided into sticks, the last stick of input data arranged in a stick will contain invalid data and will require padding. The padding locations are automatically identified according to the process described herein such that a portion of the specified slice is flagged to begin masking, which is performed until the end of slice number 7, which is the 8th slice. Processing continues at s=X xsl Iterates through -1 for the total number of slices.

[0073] The process proceeds to decision step S706, where it is determined whether the value of slice count is greater than or equal to X xsl When the slice count “s” is equal to the value of X, the mask pattern for that stick is complete and the masking process follows the “No” branch of mask generation for a subsequent mask operation performed, for example, by the mask operation module 606 (FIG. 6). xsl is compared with the value of

[0074] After the "Yes" branch, processing continues to decision step S708, where the current value of slice_count is equal to SliceID xsl The value of SliceID is compared to the xsl The value of SliceID is the slice number where masking begins. xslIf is less than the slice count value, the process follows the "Yes" branch for the slice that needs to be completely padded, i.e., masked. xsl The values ​​are all set to "1" to indicate that the corresponding slice is to be masked (S710), and the slice count is incremented by 1 (S712).

[0075] After the "No" branch, processing proceeds to decision step S714, where Valid xsl The value of L wsl is compared with the value of Valid. xsl The value of Mask is the number of elements in a given slice where masking begins. If the two values ​​are equal, processing follows the "Yes" branch for the slice or elements of the slice that should be completely padded, i.e., masked. xsl The values ​​of are all set to "1" (S716), and the process proceeds to step S720.

[0076] After the "No" branch, the process proceeds to step S718, where Mask xsl The values ​​of are set to all "0", indicating that the slice or element should not be masked.

[0077] Processing continues to decision step S720, where Valid xsl The value of is compared to a value of zero. xsl If is zero, the indicated element should not be masked, so the process follows the "Yes" branch and sets Mask wsl All values ​​are set to "0" (S722), and the slice count is incremented by 1 (S724).

[0078] After the "No" branch, processing proceeds to decision step S725, where a decision is made based on the value of s. xsl If equal to, the indicated element should not be masked, so processing follows the "No" branch and sets Mask wslAll values ​​are set to "0" (S722), and the slice count is incremented by 1 (S724).

[0079] After the "Yes" branch, processing proceeds to decision step S726, where a determination is made as to whether the inner dimension is an intra-slice dimension (wsl) or a cross-slice dimension (xsl). If the inner dimension is an intra-slice dimension, processing follows the "wsl" branch and applies a Mask to the elements in the specified range within the stick. wsl is set to "1" (S728), and Mask is applied to the remaining elements in the stick. wsl are all set to "0" (S730). The process proceeds to step S732.

[0080] After the "xsl" branch, processing proceeds to step S734, where the term "l" is set to a value of zero.

[0081] Processing continues to decision step S736, where the value of "l" is determined to be L xsl is compared with the value of L xsl If L is less than or equal to zero, which is the first set of values ​​for "l", then processing follows the "No" branch to step S732. The term "l" is an inner dimension counter, L xsl indicates the number of inner dimensions for which the inner dimension is a cross-slice dimension.

[0082] L xsl If is greater than the inner dimension count of zero, processing follows the "Yes" branch to step S738, where Mask wsl is set to "1" for all elements in the range to be masked, and then proceeds to step S740, where Mask wsl is set to all "0" for range elements in the slice that should not be masked.

[0083] The process proceeds to step S742, where the value of l is incremented by one.

[0084] Processing returns to decision step S736, where L xsl The value of L is compared to the revised value of l (from step S742), which is 0+1=1 in the first iteration. xsl If is greater than one, processing follows the "No" branch to step S732; otherwise, processing continues by calculating Mask for the specified element of the slice. wsl Return to the steps of setting the value of 1 (S738) and 0 (S740), and set the value of l to L xsl , which triggers the process to follow the "No" branch to step S732.

[0085] Processing continues down the "NO" branch to step S732 where the slice counter is incremented by one and processing returns to decision step S706. Processing continues as each slice of the selected stick is processed until the slice counter value is incremented by one. xsl , which triggers the process to follow the "No" branch and proceed to decision step S744.

[0086] Processing continues down the "no" branch to decision step S744 where the mask all (MA) value is compared to 1. If the MA value is set to 1, processing continues down the "yes" branch to set the final mask value to all "1's" (S746), thereby masking out each element that was marked as a 1 during mask patterning and terminating the process.

[0087] Following the "No" branch to step S748, xsl and Mask wsl The process ends when the final mask pattern is obtained by performing a logical OR operation on the values ​​associated with

[0088] To summarize the above process, for all 8 slices in each stick from slice 0 to slice 7, the cross-slice mask is generated as follows. For slices that should be fully padded, the cross-slice mask Mask xsl is set to 1. For all other slices, the cross-slice mask is set to 0. SliceID is the slice where masking begins. xsl For , the intra-slice mask is generated as follows: If the innermost dimension is of cross-slice type, the least significant bit of each intra-slice mask part is set to '0' and the other bits are set to '1'. The least significant bit is calculated using the following formula: Bit Significance Measure=elsz*Valid xsl If the innermost dimension is within a slice, then for all cross-slice dimension sub-slice slices, the least significant bit of each within-slice mask portion is set to '0' and the rest are set to '1'. For all other slices, each within-slice mask portion is set to '0'.

[0089] Some embodiments of the present invention are directed to masking instruction formats of less than 32 bits, such as those shown in Table 3, which derive the masking format from information provided. The example instruction format operates to mask the most significant element of a slice starting with element n. xsl is the slice number where masking starts based on n. xsl is the SliceID where masking starts xsl is the number of elements in the stick. A value of zero indicates that masking of all eight elements is not performed. The term MA (mask-all) is set to 1 to mask all elements of the stick. REGFILE[msk_src] is used as a 16-bit value to replace the elements to be masked.

[0090] [Table 3]

[0091] For example, referring to the number format FP16, X xsl =8, L xsl = 1, and the cross slice (xsl) is not innermost in the slice, i.e., the slice layout is {[X xsl ][L xsl ][L wsl ]} where "xsl" refers to the cross slice dimension and "wsl" refers to the within slice dimension. The calculation is as follows: L xsl *L wsl =8, L wsl =8 For a width of 43 elements, SliceID_Xsl=43 / 8=5, Valid_Xsl=43%8=3 (zero is interpreted as 8 (no masking))

[0092] An example pseudo program using the SAMV instruction to load a NCHW=1x1x64x43 tensor, where W=43 is the stick dimension, aligned with 64 sticks and 64 elements, and padding invalid elements with 0x3e00, is as follows: 0:Regcopy (imm=0x3e00,index=0) 1:SAMV(layout=0v110011,msk_src=0,Valid_Xsl=0b000011,SliceID_Xsl=0b101,MA=0) 2: Loop (i=0;i<64;i++) { 3: Load(mem[i]) }

[0093] An example mask pattern generated according to the pseudo program above, and an example 1 x 1 x 64 x 43 tensor are shown in Table 4.

[0094] [Table 4]

[0095] Some embodiments of the present invention are directed to on-the-fly padding of data from a first set of data to a first data volume (format referred to as a stick) for performance and code path improvement with a computer system having at least one central processing unit (CPU), at least one storage associated with the at least one CPU, at least one artificial intelligence (AI) accelerator unit (AU) associated with the at least one CPU, and a load unit (LU) for padding the first set of data while loading the first set of data from the at least one storage to the first data volume (format) (called a stick). The process may include (i) storing a set of values ​​in a register file (RF) of the LU during initialization or during run-time based on parameters associated with the first set of data; (ii) specifying a mask pattern using an instruction called SAMV (Set Active Mask and Values) for the first data volume (format); (iii) selecting a set of values ​​from the register file to replace masked elements in the first data volume (format / stick); (iv) leaving unmasked elements of the first data volume (format / stick) unchanged; (v) after occurrence of the SAMV instruction, saving the set of derived masks and associated replacement values; and (vi) applying the derived masks and associated replacement values ​​by all subsequent equivalent load instructions to pad data from the first set of data to the first data format / stick accordingly.

[0096] According to some embodiments of the present invention, the mask pattern is not directly specified but is derived from layout and pad range parameters encoded within bit fields of the SAMV instruction.

[0097] According to some embodiments of the present invention, a combination of cross-slice (xsl) masking and within-slice (wsl) masking is used to reduce the complexity of mask decoding with coarse-grained and fine-grained masks, respectively.

[0098] Some embodiments of the present invention are directed to a compact 6-bit stick data layout encoding that utilizes 2 bits for an element size indication, 3 bits for an innermost dimension length indication, and 1 bit to indicate whether the innermost dimension is cross-slice or within a slice. This supports hierarchical layouts with inner and outer dimensions within a slice. Thus, replacement of pad elements can be controlled to occur along the inner or outer dimension within a slice.

[0099] Some embodiments of the present invention specify a mask pattern based on compact layout and range parameters of the SAMV instruction.

[0100] Some embodiments of the present invention are directed to selecting a set of values ​​from a register file to replace masked elements without modifying unmasked elements.

[0101] Some embodiments of the present invention are directed to saving the derived mask after a SAMV instruction occurs and applying the derived mask and associated replacement value by all subsequent equivalent load instructions to pad the data on the fly.

[0102] According to some embodiments of the present invention, the input data may be padded with different values ​​based on different processing steps. For example, for logsoftmax in the softmax step, the padding value is the smallest representable number, while for the log step, it is 0. (Please note that the terms "logsoftmax" and / or "softmax" may be subject to trademark rights in various jurisdictions throughout the world and, to the extent such trademark rights may exist, are used only in connection with products or services appropriately indicated by the trademark.)

[0103] Some embodiments of the present invention are directed to storing the pad value in a register file (RF).

[0104] Some embodiments of the present invention add performance benefits by padding on-the-fly, without the need to store the padded data.

[0105] Some embodiments of the present invention have the following features, characteristics, and / or advantages: (i) support for 2D padding patterns while adding more flexibility with respect to data formats; (ii) when providing mixed software and hardware products, embodiments of the present invention are very likely to be more area efficient than conventional solutions; (iii) performance is improved since no extra time is added to the padding process; (iv) the compact layout of the encoding eliminates the need to store explicit data structures for the mask in memory; and (v) generic masking patterns are implicitly defined using only a few bits, which is well suited for short instruction formats. (vi) doing more than adding padding in a pre-processing step and storing the padded vector in a second memory; (vii) applying padding on-the-fly while loading data to the consumer; (viii) doing more than padding based on a padding substitution table and a mask table; (ix) synthesizing a padding mask online based on a compact layout and range parameters that fit into a short instruction; (x) the application of on-the-fly padding is not limited to convolutions and has broader applications; or (xi) reducing the complexity or area of ​​providing instructions, or any combination thereof.

[0106] Here are some useful definitions:

[0107] The present invention: The subject matter described by the term "the present invention" should not be construed as an absolute indication that it is covered by either the claims as filed or the claims that may ultimately be issued after patent prosecution. The term "the present invention" is used to help the reader get a general sense of what disclosures herein are new, but as indicated by the use of the term "the present invention", this understanding is provisional and tentative. It may be subject to change during the course of patent prosecution as relevant information evolves and the claims may be amended.

[0108] Embodiments: See definition of "present invention" above. A similar caveat applies to the term "embodiments".

[0109] or…or both (combinations thereof): inclusive, or, for example, A, B, “or C or a combination thereof” means that at least one of A or B or C is true and applicable.

[0110] User / Registrant: includes, but is not limited to, (i) a single individual human being, (ii) an artificial intelligence entity with sufficient intelligence to act as a user or registrant, or (iii) a group of related users or registrants, or any combination thereof.

[0111] Module / Sub-Module: Any set of hardware, firmware, or software, or combination thereof, operable to perform a certain function, whether the module is (i) in a single local neighborhood, (ii) distributed over a wide area, (iii) in a single neighborhood within a larger software code, (iv) located within a single software code, (v) located within a single storage device, memory, or medium, (vi) mechanically connected, (vii) electrically connected, or (viii) connected in data communication, or any combination thereof.

[0112] Computer: Any device that has the ability to process large amounts of data and / or read machine-readable instructions, including, but not limited to, desktop computers, mainframe computers, laptop computers, Field Programmable Gate Array (FPGA)-based devices, smart phones, personal digital assistants (PDAs), body worn or inserted computers, embedded device computers, and Application Specific Integrated Circuit (ASIC)-based devices.

Claims

1. 1. A method for padding data while loading data from a memory to a data consumer, comprising: receiving load instructions including padding instructions and read instructions to read input data arranged in a stick-like layout from a memory to a data consumer, the padding instructions including replacement values ​​for elements to be masked and padding parameters; deriving a mask pattern from parameters specified in the padding instruction; padding the input data by masking invalid elements of the stick layout; generating a set of data sticks including the padded data for the data consumer.

2. The method of claim 1 , wherein the parameters specified in the padding instruction include a layout parameter and a pad range parameter.

3. 2. The method of claim 1, further comprising the step of transmitting said set of data sticks to said data consumer in response to said load command.

4. 2. The method of claim 1, further comprising the step of decoding said padding instruction to determine said replacement value and said mask pattern.

5. The stick layout is separable into a predefined number of slices, The masking of the invalid elements comprises: Identifying an invalid slice in a stick, the invalid slice including an invalid element; and determining a starting slice having a portion of valid elements and a portion of invalid elements.

6. 2. The method of claim 1, wherein the padding instruction provides a data format specified in a 6-bit instruction that includes a data format of sticks, a cross-slice dimension type, and a major dimension.

7. 2. The method of claim 1, wherein the stick-like layout in which the input data is arranged is of a predetermined size that matches a single instruction multiple data (SIMD) processing capability of an accelerator that performs the deriving and padding steps.

8. 1. A computer program product comprising a computer-readable storage medium having stored thereon a set of instructions, the set of instructions, when executed by a processor, causing the processor to: receiving a load instruction including a padding instruction, the load instruction including a read instruction to read input data arranged in a stick-like layout from a memory to a data consumer, the padding instruction including replacement values ​​for elements to be masked and padding parameters; deriving a mask pattern from parameters specified in the padding instruction; padding the input data by masking invalid elements of the stick layout; and generating a set of data sticks including the padded data for the data consumer.

9. 9. The computer program product of claim 8, wherein the parameters specified in the padding instruction include a layout parameter and a pad range parameter.

10. The processor, 9. The computer program product of claim 8, further comprising: in response to said load command, transmitting said set of data sticks to said data consumer.

11. The processor, 9. The computer program product of claim 8, further comprising: decoding the padding instruction to determine the replacement value and the mask pattern.

12. said stick-like layout being separable into a predefined number of slices; Masking the invalid elements comprises: Identifying an invalid slice in a stick, the invalid slice including an invalid element; and determining a starting slice having a portion of valid elements and a portion of invalid elements.

13. 9. The computer program product of claim 8, wherein the padding instruction provides a data format specified in a 6-bit instruction that includes a data format of sticks, a cross-slice dimension type, and a major dimension.

14. 1. A computer system for padding data while loading the data from a memory to a data consumer, comprising: A processor set; a computer-readable storage medium; the set of processors is constructed, arranged, connected, or programmed, or any combination thereof, to execute program instructions stored on the computer-readable storage medium; The program instructions, when executed by the processor set, cause the processor set to: receiving a load instruction including a padding instruction, the load instruction including a read instruction to read input data arranged in a stick-like layout from a memory to a data consumer, the padding instruction including replacement values ​​for elements to be masked and padding parameters; deriving a mask pattern from parameters specified in the padding instruction; padding the input data by masking invalid elements of the stick layout; and generating a set of data sticks including the padded data for the data consumer.

15. 15. The computer system of claim 14, wherein the parameters specified in the padding instruction include a layout parameter and a pad range parameter.

16. The processor set includes:

15. The computer system of claim 14, further configured to perform a method including, in response to said load command, transmitting said set of data sticks to said data consumer.

17. The processor set includes:

15. The computer system of claim 14, further configured to perform a method including decoding the padding instruction to determine the replacement value and the mask pattern.

18. said stick-like layout being separable into a predefined number of slices; Masking the invalid elements comprises: Identifying an invalid slice in a stick, the invalid slice including an invalid element; and determining a starting slice having a portion of valid elements and a portion of invalid elements.

19. 15. The computer system of claim 14, wherein the padding instruction provides a data format specified in a 6-bit instruction that includes a data format of sticks, a cross-slice dimension type, and a major dimension.

20. 1. A computer-implemented method comprising: Decoding a padding instruction to determine replacement pad value and mask pattern padding parameters for a set of stuck input data, the padding instruction being embedded in a first load instruction of the stuck input data; deriving a mask pattern from said padding parameters; and applying the derived mask pattern and the associated replacement pad value to a subsequent load instruction to load subsequent input data, wherein the subsequent load instruction is equivalent to the first load instruction, and wherein the subsequent input data is padded during transmission to a requesting consumer.

21. 21. The method of claim 20, wherein the set of stickified input data is arranged as sticks, each stick being separable into a predefined number of slices.

22. and padding the set of stickified input data by masking invalid elements of each stick to generate a set of data sticks, wherein the masking of invalid elements of sticks includes: identifying an invalid slice in the stick, the invalid slice including an invalid element; and determining a starting slice having a portion of valid elements and a portion of invalid elements.

23. 21. The method of claim 20, wherein the replacement pad value is determined by reference to a pad value register file.

24. 21. The method of claim 20, further comprising generating a set of data sticks from the set of stickified input data for a requesting consumer, the set of data sticks being padded according to the derived mask pattern that includes the replacement pad value.

Images