Flexible access instructions for efficient access to ML data

Abstract

A flexible access instruction for efficient access to ML data. A graphics processing unit (GPU) and a method are disclosed that perform a convolution operation recast as a matrix multiplication operation. The GPU includes a register file, a processor, and a state machine. The register file stores data of an input feature map and data of a filter weight kernel. The processor performs the convolution operation on the data of the input feature map and the data of the filter weight kernel as a matrix multiplication operation. The state machine facilitates performance of the convolution operation by unrolling the data of the input feature map and the data of the filter weight kernel in the register file. The state machine includes a control register that determines movement of the data through the register file to perform the matrix multiplication operation on the data in the register file in an unrolled manner.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 128,838, filed December 21, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter disclosed herein relates to a graphics processing unit (GPU). More specifically, the subject matter disclosed herein relates to a GPU system and a method that improves the efficiency of accessing data for three-dimensional (3D) matrices, which are frequently encountered in machine learning (ML) tasks, thereby reducing storage and memory access overhead. Background Technology

[0004] Modern GPUs are used for both graphics-intensive operations and compute-intensive workloads. A GPU typically consists of a set of programmable, highly parallel compute engines and a collection of various fixed-function units. Some of these fixed-function units can operate to provide, but are not limited to, texture address generation and filtering, primitive clipping, culling, viewport transformation, merging, rasterization setup and rasterization, depth comparison, blending, and other operations.

[0005] Machine learning continues to gain popularity in consumer products and has become quite prevalent on mobile devices. From a high-level perspective, ML can be divided into training and inference tasks. Training on mobile devices can often be computationally intensive enough to run offline, such as in the cloud, or simply as background processing during off-peak hours on the mobile device. Inference, on the other hand, as an application of the ML model, runs in real-time on mobile devices and can involve large datasets and significant computation, although typically less computationally intensive than during training.

[0006] Given the significant computational demands of many ML inference operations, many System-on-Chip (SoCs) can employ dedicated Neural Processing Units (NPUs) to support a wide range of ML model types and operand formats. The intention behind using an NPU is to provide a more efficient performance / wattage computational unit than what might be achievable using a Central Processing Unit (CPU) or GPU. While this may be the case for some ML models running with certain input operands, there may be situations where the general-purpose nature of a GPU is more suitable, or where an NPU might be more appropriate when operands are not supported (e.g., 32-bit floating-point (FP32)). In some cases, most layers of an ML model might run in low precision (i.e., int8), but one layer might need to run in FP32 precision (e.g., softmax). Summary of the Invention

[0007] One example embodiment provides a graphics processing unit that may include a register file, a processor, and a state machine. The register file may store data of an input feature map and data of a filter weight kernel. The processor may perform a convolution operation on the data of the input feature map and the data of the filter weight kernel as a matrix multiplication operation. The state machine may facilitate the execution of the convolution operation by unrolling the data of the input feature map and the data of the filter weight kernel in the register file. In one embodiment, the state machine may include a control register that determines the movement of data through the register file to facilitate matrix multiplication of the data in the register file in an unrolled manner, wherein the control register may be configured based on at least one parameter of the input feature map and at least one parameter of the filter weight kernel. The state machine may receive the at least one parameter of the input feature map and the at least one parameter of the filter weight kernel from at least one instruction set architecture instruction. In another embodiment, the unrolled data of the input feature map and the unrolled data of the filter weight kernel may be unrolled based on one of the im2col technique and a memory-efficient convolution technique. In another embodiment, the state machine may further facilitate the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel in the first set of registers of the register file, while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into the second set of registers of the register file, and facilitating the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel into the second set of registers of the register file. The state machine may also repeatedly facilitate matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel until the matrix multiplication is completed by: facilitating the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel in the first set of registers of the register file while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into the second set of registers of the register file, and controlling the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel in the second set of registers of the register file while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into the first set of registers of the register file. In one embodiment, the convolution operation includes operations of one of a convolutional neural network, a recurrent neural network, a generative adversarial network, and an autoencoder.

[0008] One example embodiment provides a graphics processing unit that may include a register file, a shader data path, and a state machine. The register file may store data of an input feature map and data of a filter weight kernel. The state machine facilitates a convolution operation performed as a matrix multiplication by the shader data path by unrolling the data and storing the unrolled data in the register file, and controlling the movement of data from the register file through the shader data path to perform matrix multiplication. The unrolled data includes data of the input feature map and data of the filter weight kernel. In one embodiment, the state machine may include a control register that determines the movement of data through the register file and the shader data path to perform matrix multiplication on the data in the register file in an unrolled manner, wherein the control register may be configured based on at least one parameter of the input feature map and at least one parameter of the filter weight kernel. The state machine may receive the at least one parameter of the input feature map and the at least one parameter of the filter weight kernel from at least one instruction set architecture instruction. In another embodiment, the unrolled data of the input feature map and the unrolled data of the filter weight kernel may be unrolled based on one of the im2col technique or a memory-efficient convolution technique. In another embodiment, the state machine may further facilitate the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel in the first set of registers of the register file while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into the second set of registers of the register file, and facilitate the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel in the second set of registers of the register file. In yet another embodiment, the state machine may further repeatedly facilitate matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel until the matrix multiplication is completed by: facilitating the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel in the first set of registers of the register file while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into the second set of registers of the register file, and facilitating the matrix multiplication of the expanded data of the input feature map and the expanded data of the filter weight kernel in the second set of registers of the register file while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into the first set of registers of the register file. Convolution operations may include operations of one of convolutional neural networks, recurrent neural networks, generative adversarial networks, and autoencoders.

[0009] An example embodiment provides a method for performing a convolution operation in a graphics processing unit (GPU) as a matrix multiplication operation. The method may include: configuring registers of a state machine in the GPU based on at least one parameter of an input feature map and at least one parameter of a filter weight kernel, wherein the input feature map can be convolved by the filter weight kernel; receiving memory addresses of data from the input feature map and data from the filter weight kernel; issuing instructions from the state machine to load the unwound data of the input feature map and the unwound data of the filter weight kernel into registers in a register file of the GPU; and multiplying the unwound data of the input feature map and the unwound data of the filter weight kernel matrix-wise. The unwound data of the input feature map and the unwound data of the filter weight kernel may be unwound based on one of the im2col technique and a memory-efficient convolution technique. The registers of the state machine may be configured by: receiving the at least one parameter of the input feature map and the at least one parameter of the filter weight kernel from at least one instruction set architecture instruction; and configuring the registers of the state machine in the GPU in response to the at least one instruction set architecture instruction. In one embodiment, multiplying the expanded data of the input feature map and the expanded data matrix of the filter weight kernel may include: multiplying the expanded data of the input feature map and the expanded data matrix of the filter weight kernel in a first set of registers of the register file, while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into a second set of registers of the register file; and multiplying the expanded data of the input feature map and the expanded data matrix of the filter weight kernel in the second set of registers of the register file. In another embodiment, multiplying the expanded data of the input feature map and the expanded data matrix of the filter weight kernel may also repeatedly include the following until the multiplication operation of the expanded data of the input feature map and the expanded data matrix of the filter weight kernel is completed: multiplying the expanded data of the input feature map and the expanded data matrix of the filter weight kernel in a first set of registers of the register file, while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into a second set of registers of the register file; and multiplying the expanded data of the input feature map and the expanded data matrix of the filter weight kernel in the second set of registers of the register file, while loading the expanded data of the input feature map and the expanded data of the filter weight kernel into a first set of registers of the register file. Convolution operations can include operations from one of the following: convolutional neural networks, recurrent neural networks, generative adversarial networks, and autoencoders. Attached Figure Description

[0010] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the accompanying drawings, in which:

[0011] Figure 1AThree example input feature maps and two corresponding filter weight kernels for each input feature map are shown;

[0012] Figure 1B This shows the expansion according to the im2col algorithm. Figure 1A Example input feature maps and corresponding filter weight kernels;

[0013] Figure 2 A functional block diagram of an example embodiment of a GPU based on the subject matter disclosed herein is drawn, which improves the efficiency of accessing data for 3D matrices frequently encountered in ML tasks;

[0014] Figure 3 This is a flowchart illustrating an example method for performing convolution operations as matrix multiplication operations in a GPU, based on the topic disclosed here; and

[0015] Figure 4 An electronic device including a GPU based on the subject disclosed herein, which performs convolution operations recast as matrix multiplication, is illustrated. Detailed Implementation

[0016] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of this disclosure. However, those skilled in the art will understand that the disclosed aspects can be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail so as not to obscure the subject matter disclosed herein.

[0017] Throughout this specification, references to "an embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in an embodiment," "according to an embodiment," or "according to an embodiment" (or other phrases with similar meanings) appearing throughout this specification may not necessarily refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not to be construed as necessarily being preferred or advantageous over other embodiments. Additionally, in one or more embodiments, particular features, structures, or characteristics may be combined in any suitable manner. Furthermore, depending on the context discussed herein, singular terms may include their corresponding plural forms, and plural terms may include their corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional", "pre-determined", "pixel-specific", etc.) are sometimes used interchangeably with their non-hyphenated counterparts (e.g., "two-dimensional", "pre-determined", "pixel specific", etc.), and uppercase entries (e.g., "counter clock", "row select", "PIXOUT", etc.) are interchangeable with their non-uppercase counterparts (e.g., "counter clock", "row select", "pixout", etc.). This occasional interchangeability should not be considered inconsistent with each other.

[0018] Furthermore, depending on the context of the discussion herein, singular terms may include their corresponding plural forms, and plural terms may include their corresponding singular forms. It should also be noted that the various figures (including component diagrams) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, various waveform and timing diagrams are shown for illustrative purposes only. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Additionally, reference numerals have been repeated between figures where appropriate to indicate corresponding and / or similar elements.

[0019] The terminology used herein is for the purpose of describing some exemplary embodiments only and is not intended to limit the claimed subject matter. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the terms “first,” “second,” etc., are used as labels for nouns preceding them and do not imply any kind of ordering (e.g., spatial, temporal, logical, etc.) unless expressly defined as such. Furthermore, the same reference numerals may be used in two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, this usage is merely for simplicity of illustration and ease of discussion; it does not imply that the construction or architectural details of such components or units are identical in all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the exemplary embodiments disclosed herein.

[0020] It will be understood that when an element or layer is referred to as being on, "connected to," or "coupled to" another element or layer, it can be directly on, directly connected to, or directly coupled to that other element or layer, or there can be intervening elements or layers. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element or layer, there are no intervening elements or layers. The same numbers always refer to the same element. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0021] As used herein, the terms “first,” “second,” etc., are used as labels for nouns preceding them and do not imply any kind of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same reference numerals may be used in two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, this usage is merely for the sake of simplicity of illustration and ease of discussion; it does not imply that the construction or architectural details of such components or units are identical in all embodiments, or that such commonly referenced parts / modules are the only way to implement some of the example embodiments disclosed herein.

[0022] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this subject pertains. It will also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant field, and will not be interpreted in an idealized or overly formalized sense, unless expressly defined herein.

[0023] As used herein, the term "module" means any combination of software, firmware, and / or hardware configured to provide the functionality associated with the module described herein. For example, software may be embodied as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any implementation described herein may, for example, individually include or in any combination include assemblies, hardwired circuitry, programmable circuitry, state machine circuitry, and / or firmware storing instructions to be executed by programmable circuitry. Modules may be embodied collectively or individually as circuitry forming part of a larger system, such as, but not limited to, integrated circuits (ICs), system-on-a-chip (SoCs), components, etc.

[0024] The subject matter disclosed herein relates to a GPU system and a method that improves the efficiency of accessing data (such as input feature map data and filter kernel data) frequently encountered in ML tasks for 3D matrices, thereby reducing storage and memory access overhead. Therefore, the subject matter disclosed herein can include both hardware and software modules that collaborate to improve the efficiency of the main function used for CNN evaluation (i.e., refactoring convolution operations into matrix multiplication operations). The subject matter disclosed herein can also be extended to RNNs (Recurrent Neural Networks), GANs (Generative Adversarial Networks), and autoencoders. In some cases, the subject matter disclosed herein can also improve the efficiency of fixed-function units (such as, but not limited to, Load Memory Units (LSUs) and Local Memory Units).

[0025] One example embodiment disclosed herein provides an apparatus and a method for improving the efficiency of convolution operations by recasting convolution operations into matrix multiplication operations. After process initialization, data can be prefetched from main memory and expanded in the GPU's register file using techniques such as im2col or Memory Efficient Convolution (MEC) to recast the convolution operation into a matrix multiplication operation. The address stream can be generated by a state machine that effectively reduces the number of memory fetches required for providing data in the register file and removes significant address calculations from the shader program. The state machine can also generate appropriate cache hints for system memory. Output formats such as, but not limited to, NHWC (batch, height, width, channels) can be provided by both the apparatus and method disclosed herein. Therefore, in cases where ML inference evaluation is targeted at the GPU, the subject matter disclosed herein can reduce the memory footprint and memory bandwidth requirements for ML inference evaluation.

[0026] The subject matter disclosed herein is both flexible and scalable enough to combine operations and support currently available or potentially future memory packing and caching efficiency techniques. Various parameters (both runtime and synthesis time) allow the state machine of the system disclosed herein to evolve with state-of-the-art algorithms available for memory packing and caching efficiency techniques.

[0027] The topics disclosed here provide several instruction set architecture (ISA) level instructions that can be added to shaders to provide efficient access to input feature map data and filter weight data. For example, the fetch instructions, referred to herein as flexible access instructions, can use hardware outside the GPU's shader core to provide efficient fetching and storage of ML input feature map data and weight data to the GPU's register file, as well as the return of the data for computation. Additionally, ISA level instructions can be used to initialize various address computations that can be executed to facilitate loading data into the shader core, allowing convolution operations to be efficiently refactored into matrix multiplication operations.

[0028] Another example embodiment may include shader software that initializes and configures the hardware, subsequently using the hardware advantageously to access CNN data, perform convolutions converted to matrix multiplication, and return the results to memory; and be further optimized to combine the layers of the model and advantageously perform arithmetic. Yet another example embodiment may include driver software that coordinates operations, provides memory management and an interface to application-level code (e.g., Neural Network Application Programming Interface (NNAPI)) or other intermediate-level tools.

[0029] A common operation encountered during inference evaluation is the evaluation of convolutions in convolutional neural networks (CNNs). However, GPUs may not be well-suited for direct convolution evaluation. In contrast, GPUs may excel at matrix multiplication and can achieve fairly high efficiency for such tasks. Therefore, the subject presented here refactors convolution evaluation into matrix multiplication, making such evaluation more suitable for the capabilities of GPUs.

[0030] Figure 1A Three example input feature maps 101-103 are shown. Input feature maps 101-103 can be three channels (RGB) of an image. For this example, there are two filter weight kernels that will be applied to each input feature map 101-103. More specifically, filter weight kernel 101... a and 101 b This will be applied to the input feature map 101. Filter weight kernel 102 a and 102 b This will be applied to the input feature map 102, and the filter weight kernel 103. a and 103 b This will be applied to the input feature map 103. It should be understood that the data for the input feature map can be, but is not limited to, image data, audio data, system data, manufacturing data, signal propagation data, sales and marketing data, financial data, medical data, weather data, scientific data, traffic data, and language data. Figure 1B This shows the expansion according to the im2col algorithm. Figure 1A Example input feature maps 101-103 and corresponding filter weight kernels. Figure 1A and Figure 1B The numbers and filter weight kernels shown in the matrices of both are intended to indicate element identification.

[0031] In execution Figure 1B When performing a [2x27]x27x9 matrix product, the result (not shown) is an augmented matrix, where each element is the numerical equivalent of applying the convolution kernel to the input feature map. Therefore, convolution can be recast as matrix multiplication. However, this naive approach of recasting convolution operations as matrix multiplication can lead to an increase in memory space and overhead for storing the input feature maps of approximately ~2x to ~9x.

[0032] It should be noted that Figure 1B The [27x9] matrix 101 contains a considerable amount of redundancy in its elements. The subject matter disclosed herein provides a technique that transmits the [27x9] matrix to the shader core of the GPU during matrix multiplication evaluation without extracting or storing the redundant information.

[0033] While convolution operations are a fundamental building block of machine learning, they are not well-suited for GPUs. System-on-a-Chip (SoC) can include a dedicated, fixed-function NPU that offers more efficient performance / watt throughput for convolutions than GPUs. However, NPUs may not support all learning models and data types (FP32), and SoCs may rely on general-purpose engines (such as GPUs) to implement such machine learning models. There may be situations where the SoC lacks an NPU or uses both NPUs and GPUs for load balancing. To enable GPU support for such learning models, convolutions could be recast as matrix multiplications, potentially increasing memory footprint and data transfer overhead by ~2x to ~9x. Therefore, saving bandwidth by reducing the memory footprint associated with convolution operations can be a desirable approach.

[0034] Figure 2 A functional block diagram of an example embodiment of a GPU 200 based on the subject matter disclosed herein is shown, which improves the efficiency of accessing data 3D matrices frequently encountered in ML tasks. The GPU 200 includes a register file 201, multiple multiplexers 202, multiple hierarchical registers 203, a shader data path 204, a bypass 205 to the multiplexers 202 and hierarchical registers 203, a state machine 206, and a read address multiplexer 207. The register file 201, multiple multiplexers 202, multiple hierarchical registers 203, shader data path 204, bypass 205, and read address multiplexer 207 can be configured as follows: Figure 2 The layout and interconnections are shown. Shader data path 204 may represent a processing device, such as, but not limited to, a shader core. State machine 206 may include control register 208 and fetch and return address generation logic 209.

[0035] Instruction decoder 210 can decode each of the existing ISA instructions and the instruction set architecture (ISA) instructions shown in Table 1-4. These instructions are used to configure state machine 206 and control operations when converting or recasting convolution operations into matrix multiplication operations. Control register 208 can receive parameter values ​​from the host device (not shown) of control fetch and return address generation logic 209 to generate addresses for elements of the input feature map and filter weight kernel, thereby affecting the unrolling of data as needed to convert convolution operations into matrix multiplication operations and generating return addresses for the resulting data.

[0036] An example implementation may include the following ISA instructions:

[0037] Table 1. Initialization instructions for inputting feature map data

[0038]

[0039]

[0040] Table 2. Initialization instructions for inputting filter weight data

[0041]

[0042] Table 3. Initialization commands used for output result data

[0043] Set the base address of the output data

[0044] Table 4. Instructions for operations during matrix multiplication

[0045]

[0046] The operation of GPU 200 consists of three phases: initialization, fetching, and streaming. The initialization phase can be initiated by a thread of the computation kernel, which initializes register 208 of state machine 206 based on the convolution computation problem at hand. For example, using the above... Figure 1A The example inputs feature map data and filter weight data, with the X and Y dimensions each initialized to 5 using the "Set X Dimension" and "Set Y Dimension" instructions for image data, respectively. The number of components is initialized to 3 using the "Set Number of Components" instruction. The stride is initialized to 1 using the "Set Stride" instruction for accessing the image data, and the component depth is initialized to 8 bits using the "Set Component Depth" instruction for image data. For filter weights, the number of filters is initialized to 2 using the "Set Number of Filters" instruction. For two filters, the X and Y dimensions are each initialized to 3 using the "Set X Dimension" and "Set Y Dimension" instructions for filter data, respectively. Other parameters can also be set for both the input feature map data and the filter weight data. The memory addresses where input data is encountered and where return data is written arrive from the host (not shown) and are passed to control register 208. State machine 206 has not actually performed any extractions yet.

[0047] Once all necessary parameters are set and the state of state machine 206 is established, the extraction phase and matrix multiplication begin by state machine 206 issuing the FETCH_ML instruction to load data from n registers into register file 201 containing expanded image feature map data, while adhering to the memory footprint of the original input image. In other words, instead of using a [9×27] matrix of input feature map data from register file 201, a [5×15] matrix of input feature map data can be used, which corresponds to performing an on-the-fly expansion of the input feature map data and the corresponding filter weight data, such as... Figure 1B As shown, this results in only about 30% of the data being accessed and cached, compared to previously deploying data on the host side.

[0048] During the fetch phase, the FETCH_ML instruction can operate as if loading data from the memory subsystem into the DMA engine within the register set (not shown) in register file 201. Synchronization can be maintained similarly to any conventional load instruction to ensure that subordinate arithmetic instructions do not begin before data is loaded into register file 201. Matrix multiplication then proceeds normally.

[0049] Once the sequence begins, the streaming phase occurs. During the streaming phase, arithmetic operations can be performed on some registers while other registers are being loaded with new input data and weights, thus creating a streaming engine.

[0050] By intelligently adjusting the size of the matrix to be computed by dividing the input image into hierarchical blocks, the memory system cache can be best utilized to maximize local cache consistency and avoid excessive missing of data that was recently in the cache but is still more recently overwritten. State machine 206 can be configured to provide appropriate cache hints and, if supported by the memory system, to provide requests for preloaded data.

[0051] Another example embodiment may include hardware that can automatically track subsequent arithmetic operations and the cutoff of subsequent operations to initiate subsequent fetches, thereby saving instruction slots for issuing further FETCH_ML instructions. The hardware of state machine 206 can also be configured to help efficiently compute addresses during the storage of the computed filtered data, thereby reducing the total instruction count. The hardware of state machine 206 can vary to accommodate various access modes, including, for example, the im2col algorithm and MEC algorithm, as well as others. Different numbers of hardware can be instantiated depending on the situation to allow for different cases of directly processing the input feature map and filter weight kernel. This feature can provide different design points regarding area / power / performance tradeoffs. In general, the hardware of state machine 206 can have sufficient flexibility to handle various output tensor return formats and will be able to generate address sequences for NHWC, such as:

[0052] offset_nhwc(n,c,h,w)=n*HWC+h*WC+w*C+c

[0053] Figure 3This is a flowchart of an example method 300 for performing convolution operations as matrix multiplication operations in a GPU, based on the subject matter disclosed herein. The process begins at 301. At 302, register 208 of state machine 206 is initialized based on the parameters of the input feature maps and the parameters of the filter weight kernels. Initialization instructions included in Tables 1 and 2 can be used to initialize register 208. In one embodiment, the input feature map parameters may involve multiple input feature maps and / or the filter weight kernel parameters may involve multiple filter weight kernels. In another embodiment, the base address of the output data can also be initialized using the initialization instructions included in Table 3.

[0054] At 303, the addresses of the input feature maps(multiple) and filter weight kernel(s) data are received from the host device. At 304, in response to the FETCH_ML instruction (Table 4), the expanded input feature map data and expanded filter weight kernel data are loaded into registers in the GPU's register file. State machine 206 controls the data flow from register file 201 through shader data path 204 to the destination register in register file 201 by performing address calculations and controlling load / store operations. At 305, the expanded input feature map data and expanded filter weight kernel data are matrix multiplied. In one embodiment, while the second set of registers in register file 201 is being loaded with input feature map data and filter weight data, the first set of registers in register file 201 can be used by the processor associated with shader data path 204 to perform matrix multiplication. After the second set of registers is loaded, while the first set of registers is being loaded with new input feature map data and filter weight data, the data in the second set of registers can be used to perform matrix multiplication. The process ends at 306 when the matrix multiplication operation is complete.

[0055] Although the topics disclosed herein have been described in conjunction with CNNs, it should be understood that the systems and methods disclosed herein are equally applicable to RNNs, GANs, and autoencoders. RNNs and GANs also use convolutional operations as fundamental building blocks, which can be recast into matrix multiplication operations. Furthermore, more generally, the systems and methods disclosed herein can be extended to allow for a variety of other access patterns of read and write operands independent of machine learning algorithms.

[0056] Figure 4An electronic device 400 is illustrated, comprising a GPU according to the subject matter disclosed herein, which performs convolution operations recast as matrix multiplication. The electronic device 400 can be used in, but is not limited to, computing devices, personal digital assistants (PDAs), laptop computers, mobile computers, network tablets, cordless phones, mobile phones, smartphones, digital music players, or wired or wireless electronic devices. The electronic device 400 may include a controller 410, input / output devices 420 (such as, but not limited to, keyboards, displays, touchscreen displays, cameras, and / or image sensors), memory 430, interface 440, GPU 450, and imaging processing unit 460, all coupled to each other via a bus 470. According to the subject matter disclosed herein, the GPU 450 can perform convolution operations recast as matrix multiplication. The controller 410 may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, etc. The memory 430 may be configured to store command codes or user data to be used by the controller 410.

[0057] Electronic device 400 and its various system components may include image processing unit 460. Interface 440 may be configured to include a wireless interface configured to transmit data to or receive data from a wireless communication network using RF signals. Wireless interface 440 may include, for example, an antenna. Electronic System 400 can also be used for communication interface protocols of communication systems (such as, but not limited to, Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), North American Digital Communications (NADC), Extended Time Division Multiple Access (E-TDMA), Wideband CDMA (WCDMA), CDMA2000, Wi-Fi, Municipal Wi-Fi (Muni Wi-Fi), Bluetooth, Digital Enhanced Cordless Telecommunications (DECT), Wireless Universal Serial Bus (Wireless USB), Fast Low Latency Access with Seamless Handover (Flash-OFDM), IEEE 802.20, General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, WiMAX Advanced, Universal Mobile Telecommunications Service - Time Division Duplex (UMTS-TDD), High Speed ​​Packet Access (HSPA), Evolved Data Optimized (EVDO), Long Term Evolution - Advanced (LTE-Advanced), Multichannel Multipoint Distribution Service (MMDS), 5G, etc.).

[0058] The embodiments of the subject matter and operation described in this specification can be implemented as digital electronic circuits, or as computer software, firmware, or hardware (including the structures disclosed in this specification and their equivalents), or a combination thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs (i.e., one or more modules of computer program instructions) encoded on a computer storage medium for operation by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for operation by the data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof, or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media may also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or may be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). Furthermore, the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0059] While this specification may contain numerous specific implementation details, these details should not be construed as limiting the scope of any claimed subject matter, but rather as descriptions of features specific to particular embodiments. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, or even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0060] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or to perform all shown operations to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments; it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0061] Therefore, specific embodiments of the subject matter have been described herein. Other embodiments are within the scope of the appended claims. In some cases, the actions set forth in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific order or sequential sequence to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

[0062] Those skilled in the art will recognize that the innovative concepts described herein can be modified and altered across a wide range of applications. Therefore, the scope of the claimed subject matter should not be limited to any specific exemplary teachings discussed above, but rather defined by the appended claims.

Claims

1. A graphics processing unit (GPU) comprising: a register file storing data of an input feature map and data of a filter weight kernel; a processor to perform a convolution operation on the data of the input feature map and the data of the filter weight kernel as a matrix multiplication operation; and a state machine to facilitate performance of the convolution operation by unrolling the data of the input feature map and the data of the filter weight kernel in the register file.

2. The graphics processing unit of claim 1, wherein the state machine includes a control register that determines movement of data through the register file to facilitate performance of the matrix multiplication operation on the data in the register file in an unrolled manner, the control register configured based on at least one parameter of the input feature map and at least one parameter of the filter weight kernel.

3. The graphics processing unit of claim 2, wherein the state machine receives the at least one parameter of the input feature map and the at least one parameter of the filter weight kernel from at least one instruction set architecture (ISA) instruction.

4. The graphics processing unit of claim 1, wherein the unrolled data of the input feature map and the unrolled data of the filter weight kernel are unrolled based on one of an im2col technique and a memory efficient convolution (MEC) technique.

5. The graphics processing unit of claim 1, wherein the state machine further facilitates matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in a first set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel to a second set of registers of the register file, and facilitates matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in the second set of registers of the register file.

6. The graphics processing unit of claim 5, wherein the state machine further repeatedly facilitates the matrix multiplication operation on the unrolled data of the input feature map and the unrolled data of the filter weight kernel until the matrix multiplication is completed by facilitating matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in the first set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel to the second set of registers of the register file, and facilitating matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in the second set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel to the first set of registers of the register file.

7. The graphics processing unit of claim 1, wherein the convolution operation includes operations of one of a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), and an autoencoder.

8. A graphics processing unit (GPU) comprising: a register file storing data of an input feature map and data of a filter weight kernel; a shader data path; and a state machine to facilitate performance of the convolution operation by unrolling the data of the input feature map and the data of the filter weight kernel in the register file. a state machine facilitates a convolution operation to be performed as a matrix multiplication operation by the shader data path by unrolling data of the input feature map and data of the filter weight kernel and storing the unrolled data in the register file and controlling movement of data from the register file and through the shader data path to perform the matrix multiplication operation, wherein the unrolled data includes the data of the input feature map and the data of the filter weight kernel.

9. The graphics processing unit of claim 8, wherein the state machine includes a control register that determines movement of data through the register file and the shader data path to perform the matrix multiplication operation on data in the register file in an unrolled manner, the control register configured based on at least one parameter of the input feature map and at least one parameter of the filter weight kernel.

10. The graphics processing unit of claim 9, wherein the state machine receives the at least one parameter of the input feature map and the at least one parameter of the filter weight kernel from at least one instruction set architecture (ISA) instruction.

11. The graphics processing unit of claim 8, wherein the unrolled data of the input feature map and the unrolled data of the filter weight kernel are unrolled based on one of an im2col technique or a memory efficient convolution (MEC) technique.

12. The graphics processing unit of claim 8, wherein the state machine further facilitates matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in a first set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel to a second set of registers of the register file and matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in the second set of registers of the register file.

13. The graphics processing unit of claim 12, wherein the state machine further repeatedly facilitates the matrix multiplication operation on the unrolled data of the input feature map and the unrolled data of the filter weight kernel until the matrix multiplication is completed by facilitating matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in the first set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel to the second set of registers of the register file and matrix multiplication of the unrolled data of the input feature map and the unrolled data of the filter weight kernel in the second set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel to the first set of registers of the register file.

14. The graphics processing unit of claim 8, wherein the convolution operation includes operations of one of a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), and an autoencoder.

15. A method of performing a convolution operation as a matrix multiplication operation in a graphics processing unit (GPU), comprising: configuring registers of a state machine in the GPU based on at least one parameter of an input feature map and at least one parameter of a filter weight kernel, wherein the input feature map is convolved by the filter weight kernel; receiving memory addresses for data of the input feature map and data of the filter weight kernel; issuing instructions by the state machine to load the unrolled data of the input feature map and the unrolled data of the filter weight kernel into registers of a register file of the GPU; and multiplying the unrolled data of the input feature map and the unrolled data of the filter weight kernel matrix.

16. The method of claim 15, wherein the unrolled data of the input feature map and the unrolled data of the filter weight kernel are unrolled based on one of an im2col technique and a memory efficient convolution (MEC) technique.

17. The method of claim 15, wherein configuring registers of the state machine in the GPU further comprises: receiving the at least one parameter of the input feature map and the at least one parameter of the filter weight kernel from at least one instruction set architecture (ISA) instruction; and configuring registers of the state machine in the GPU in response to the at least one instruction set architecture instruction.

18. The method of claim 15, wherein multiplying the unrolled data of the input feature map and the unrolled data of the filter weight kernel matrix comprises: multiplying the unrolled data of the input feature map and the unrolled data of the filter weight kernel matrix in a first set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel into a second set of registers of the register file; and multiplying the unrolled data of the input feature map and the unrolled data of the filter weight kernel matrix in the second set of registers of the register file.

19. The method of claim 18, further comprising, repeatedly until the matrix multiplication operation on the unrolled data of the input feature map and the unrolled data of the filter weight kernel is complete: multiplying the unrolled data of the input feature map and the unrolled data of the filter weight kernel matrix in the first set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel into the second set of registers of the register file; and multiplying the unrolled data of the input feature map and the unrolled data of the filter weight kernel matrix in the second set of registers of the register file while loading the unrolled data of the input feature map and the unrolled data of the filter weight kernel into the first set of registers of the register file.

20. The method of claim 15, wherein the convolution operation comprises operations of one of a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), and an autoencoder. ​

Images