Activation Memory Mapping for Convolutional Neural Network Implementation

Abstract

A memory controller circuit for mapping data of a convolutional neural network to a physical memory is disclosed, the memory controller circuit comprising a receiving unit for receiving a selected parameter value and a mapping unit for mapping pixel values ​​of a layer of the convolutional neural network to memory words of the physical memory according to one of a plurality of mapping schemes, the mapping being dependent on the value of the received selected parameter value.

Description

[Technical Field]

[0001] The present invention relates generally to memory controller circuits, and more particularly to memory controller circuits for mapping convolutional neural network data to physical memory. The present invention further relates to methods and computer program products for mapping convolutional neural network data to physical memory. [Background technology]

[0002] Artificial intelligence (AI) long ago emerged from the research community, but its use is becoming increasingly commonplace in enterprise computing environments. Today, AI concepts are already being incorporated into corporate applications and commercial systems (e.g., facial recognition). While various AI algorithms exist, one prominent form involves convolutional neural networks (CNNs). These are often implemented as deep neural networks with multiple hidden layers between the node input layer and the node output layer. Numerous matrix / vector transformations may be required to control the CNN's function, self-organization, or both. Specifically, when convolutions are performed in a pipelined manner in each cycle—for example, by shifting a relatively small convolution kernel matrix over a larger-valued matrix—a new row of activations (i.e., the output values ​​of the activation functions of individual nodes in the CNN) equal to the size of the convolution kernel must be read from memory. Therefore, from the first cycle to the second cycle, activations must be retrieved from their respective memory locations and loaded into the processing unit. The matrix multiplication method is often as follows: the component values ​​of the vector, e.g., activation values ​​or activations, are read row-wise, multiplied with the respective matrix values, and then stored column-wise in memory, e.g., as part of the output feature map. Depending on the bitwise configuration of the values ​​in the vector, certain addressing options may exist, which may be used to store these values ​​in memory that is insufficient for high-performance computing. Therefore, the algorithms used may be too slow to use the results of the respective CNNs in real-time applications.

[0003] There are already some documents dealing with memory access optimization. For example, U.S. Patent Publication No. 2019 / 0187963 A1 describes a method for configuring a system-on-chip (SoC) for executing a CNN process including a CNN layer, the method including: determining, for each schedule, memory access amount information describing how many memory accesses are required; expressing the memory access amount information as a relationship describing data reusability; combining the relationship with the cost of writing to and reading from an external memory; executing the memory access information; and determining memory allocations of the SoC's on-chip memory for input feature maps and output feature maps.

[0004] Furthermore, U.S. Patent Publication No. 2019 / 0102671 A1 discloses a CNN accelerator, which includes a CNN circuit for performing multiple layers of CNN computation, where the multiple layers receive input features according to an input feature map and a weight matrix for each output feature, and the output of a first layer provides the input for a next layer; and a mapping circuit for accessing a three-dimensional input matrix stored as a Z-priority matrix.

[0005] However, publications in the art do not describe an optimized method for accessing and storing vector element values ​​in a manner for fast read and write processes without the need to mask and unmask specific bit regions within consecutive memory words.

[0006] Therefore, there will be a need to provide flexible usage of memory resources for the use of CNN operations, especially ways to access activation function output values ​​and write them back to memory to support fast matrix / vector operations. Summary of the Invention

[0007] According to one aspect of the present invention, there is provided a memory controller circuit for mapping data of a convolutional neural network to a physical memory, the memory controller circuit may include a receiving unit for receiving a selected parameter value and a mapping unit for mapping pixel values ​​of one layer of the convolutional neural network to memory words of the physical memory according to one of a plurality of mapping schemes, the mapping being performed in response to the received selected parameter value.

[0008] According to another aspect of the present invention, there is provided a method for mapping data of a convolutional neural network to a physical memory, the method including receiving selected parameter values ​​and mapping pixel values ​​of a layer of the convolutional neural network to memory words of the physical memory according to one of a plurality of mapping schemes, the mapping corresponding to the received selected parameter values.

[0009] The proposed memory controller circuit for mapping convolutional neural network data to physical memory may provide many advantages, technical effects, contributions or improvements, or combinations thereof.

[0010] In general, how variables are stored in words of a computer system's physical memory is controllable. Such selectable and controllable mapping schemes can vary from application to application. These schemes fundamentally differ in how variables, i.e., bit values, can be stored in physical memory. A variable can always start at the beginning of a physical memory word or can span two words, with the first portion of the variable's bits stored in the first addressable memory word and the second portion of the variable's bits stored in the second addressable memory word adjacent to the first. Furthermore, a physical memory word can contain at most one variable, or it can contain multiple variables, with all the bits of one variable stored in one addressable memory word and all the bits of the second variable stored adjacent to the bits of the first variable in the same memory word.

[0011] Furthermore, the method for aligning variables to word boundaries can be configured externally to the memory controller circuitry and can also be dynamically selected at any time by specific parameters that can be received by the memory controller circuitry. On the other hand, the selection variables can be part of the variables of the convolutional neural network, e.g., hyper-parameters. Hence, the selection variables can be part of the meta-parameters of a layer of the complete neural network.

[0012] This flexible and adaptive alignment of variables and physical memory words allows for a balance between optimal usage of available memory and optimal performance of the CNN, e.g., for high data throughput. Furthermore, it should be noted that the concepts proposed here are not only useful for CNNs, but also for any other form of computation that involves matrix operations, especially matrix-vector multiplication.

[0013] Additionally, the proposed memory controller circuit and associated method can be used in any digital computing environment, but can also be used in crossbar arrays of computational memory devices that can store more than one bit per physical storage cell, typically using phase change memory units (PCM). Additionally, the proposed memory controller circuit can be directly integrated with digital processing devices or storage devices, or both, as well as with crossbar arrays of memory devices.

[0014] Below, additional embodiments of the inventive concepts are described that are applicable to memory controller circuits and related methods.

[0015] According to an interesting embodiment of the memory controller circuit, the mapping scheme can be an intra-word adjacent mapping scheme (IWAP), whereby the bit values ​​of different pixel value parts are mapped contiguously into one physical memory word. Therefore, the digital representation of the output of the activation function, i.e., the pixel value, is stored in contiguous memory words that span across word boundaries. This can represent the most compact format for storing the resulting feature maps of the intermediate results of a convolutional neural network.

[0016] According to another possible embodiment of the memory controller circuit, the mapping scheme can be a kernel word interleaving scheme (KLIP). This allows only the bit values ​​of different pixel values ​​to be mapped to the physical memory so that at most a number of pixels equal to the kernel size of the convolutional neural network operation can be contiguously mapped to adjacent memory words. Therefore, some space is left free in the memory, specifically, bits related to the edge of the required size, i.e., bits required for one dimension (i.e., one row of one column) of the kernel matrix for the convolution calculation. This version may not be as compact as the IWAP scheme. However, read and write operations can be faster compared to the IWAP scheme.

[0017] According to another useful embodiment of the memory controller circuit, the mapping scheme can be a powered word interleaving scheme (PLIP). The bits of a pixel value can be mapped so that a memory word contains at most one pixel's bit value. The remaining bits of the memory word can be left empty. However, if the bit length of the pixel value is greater than the physical memory word size, more than one word of physical memory can be used to store the pixel value. This storage method or mapping scheme for CNN pixel values ​​can represent the most efficient one in terms of read and write speeds. However, this storage method can require the most physical memory words.

[0018] According to another embodiment of the memory controller circuit, the selected parameter values ​​can be included in a set of parameter values ​​for a layer of a convolutional neural network, which can include, in addition to the selected parameter values, any data describing variables of the CNN, such as the number of nodes in a particular layer, the number of layers, activation functions, weight values, identifiers of connections to nodes in other layers, and other subsets of metadata parameter values ​​(or hyper-parameters of the CNN).

[0019] According to another embodiment of the memory controller circuit, the selected parameter values ​​can be independent of the parameter values ​​of the layers of the convolutional neural network. Therefore, the mapping scheme can be freely selected, and the mapping scheme used can be dynamically changed after each matrix-vector operation. This allows for a high degree of flexibility in the usability of the proposed concept. Therefore, this mapping scheme can also be used outside the scope of convolutional neural networks. Therefore, the mapping scheme of any variable to physical memory words can be changed at any time, completely independent of the application context.

[0020] According to an additional embodiment of the memory controller circuit, the physical memory can be selected from the group including static random access memory, latch-based memory, and dynamic random access memory. These types of word-addressable memory units can represent typical sets of storage used in modern computing. However, the concepts proposed herein can also be applied to other forms of word-oriented storage units (e.g., solid-state disks).

[0021] According to another embodiment of the memory controller circuit, the memory controller circuit can be integrated with the physical memory. Therefore, both components can be integrated on a chip and in silicon on a single die. This can result in a smaller physical footprint for the associated integrated circuit.

[0022] In another embodiment of the memory controller circuit, the physical memory that stores the input values ​​of the computational memory can be integrated with the computational memory. Typically, the computational memory design includes input and output latches. The concept proposed here for mapping pixel values ​​of CNNs can also be applied in a fully integrated on-chip form.

[0023] According to a further enhanced embodiment of the memory controller circuit, the physical memory that stores the digital memory input values, e.g., a regular RAM, can be integrated with a digital processing unit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or other dedicated hardware circuit, e.g., an AI accelerator.

[0024] Additionally, embodiments may include related computer program products accessible from a computer usable or computer readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable or computer readable medium may be any apparatus that includes means for storing, communicating, propagating, or transporting a program for use by or in connection with an instruction execution system, apparatus, or device. [Brief explanation of the drawings]

[0025] It should be noted that embodiments of the present invention are described with reference to different embodiments. Specifically, some embodiments are described with reference to method-type claims, and other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will understand from the above and following description that, unless otherwise specified, any combination of features belonging to one type of embodiment and any combination between features relating to different embodiments, in particular, combinations between features of method-type claims and features of apparatus-type claims, are considered to be disclosed in this specification.

[0026] The above-defined and further aspects of the present invention will be apparent from and will be elucidated with respect to the example embodiments described hereinafter, to which the present invention is not limited.

[0027] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0028] [Figure 1] 1 illustrates a block diagram of one embodiment of a memory controller circuit of the present invention for mapping convolutional neural network data to physical memory. [Figure 2] FIG. 1 shows a block diagram of one embodiment of the operations in a CNN, including feature maps, kernel matrices, and associated outputs. [Figure 3] 1 shows a block diagram of an idealized embodiment of the concepts proposed herein. [Figure 4] FIG. 1 illustrates a block diagram of one embodiment of an IWAP schema for mapping pixel values ​​to physical memory. [Figure 5] FIG. 1 shows a block diagram of one embodiment of a KLIP schema for mapping pixel values ​​to physical memory. [Figure 6] 1 shows a block diagram of one embodiment of a PLIP schema for mapping pixel values ​​to physical memory. [Figure 7]1 shows a block diagram of one embodiment of the inventive method for mapping convolutional neural network data to physical memory. [Figure 8] 1 illustrates one embodiment of a computing system including a memory controller of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] In the context of this description, the following conventions, terms or expressions, or combinations thereof, are used:

[0030] The term "memory controller circuitry" may refer to integrated electronic circuitry that controls the function of physical memory, typically in the form of semiconductor devices, and the flow of data between the physical memory and the CPU. Memory controllers may also be used to control the refresh cycles of dynamic RAM devices. However, in the context of this specification, the focus is on how variables are stored in and retrieved from physical memory, in relation to memory word size and memory word organization.

[0031] The term "mapping," as used herein, may refer to the activity of storing or retrieving a variable whose length in bits differs from the physical memory word size so that it is aligned with a word in physical memory.

[0032] The term "convolutional neural network" (CNN) can refer to an artificial neural network with an input layer of nodes, an output layer of nodes, and multiple hidden layers of nodes. A characteristic of CNNs is that mathematical convolution operations are performed to determine the variables from one feature map to the next in the network of nodes.

[0033] The term "physical memory" may refer to a conventional semiconductor storage device, e.g., a semiconductor memory chip such as a DRAM or SRAM, that consists of a single addressable memory word consisting of a predetermined fixed number of bits per address.

[0034] The term "selection parameter value" herein may refer to the value of a selection variable of a small number of bits, e.g., 2 bits, for selecting a mapping scheme between the output variable of an activation function of one of the nodes of the different layers of a neural network and the way this variable is stored in physical memory in terms of alignment with physical memory word boundaries.

[0035] The term "pixel value," or pixel for short, can refer to the output variable of one of the nodes of a neural network. In general, the term pixel value can be used herein as the output variable of a node's activation function. Therefore, the combined output of a layer in a neural network can be described as as many pixels as there are nodes available in the associated layer of the neural network.

[0036] The term "memory word" or "physical memory word" may refer to the smallest directly addressable unit of physical memory, i.e., the associated semiconductor memory device.

[0037] The term "different pixel values" can refer to different variations in the output values ​​of different activation functions of different nodes of a CNN.

[0038] The term "intra-word adjacent mapping scheme" or "intra-word adjacent placement" (IWAP) can refer to a first mapping scheme or method of a variable relative to the word boundaries of the underlying word organization in physical memory. If a variable requires fewer or more bits than the word length of physical memory, the first portion of the bits required for that variable is stored in one memory word, and the remaining portion of the bits is stored in the next memory word. The bits of the next variable are stored immediately after the end of the bits of the first variable and are contiguous with the bit string of the first variable (i.e., the bits of the two variables are stored immediately adjacent to each other, e.g., starting and ending in the middle of a memory word).

[0039] The term "kernel word interleaving scheme" or "kernel word interleaving" (KLIP) can refer to a second mapping scheme or method of variables. Essentially, the resulting variables (i.e., pixels) of a convolution kernel operation, moved across a feature map, can be stored as values ​​of the next feature map in the most compact way (i.e., immediately adjacent to each other). Therefore, each new vector can start in a new word of physical memory.

[0040] The terms "active word interleaving scheme" or "pixel-level interleaving mapping" or "pixel-level interleaving placement" (PLIP) can refer to a third mapping scheme or method for variables. Here, each variable is generally stored in one physical memory word because the number of required bits is generally smaller than the bits available in a physical memory word. However, the number of bits required for a variable value can be greater than the available bits and physical memory word. In this case, the variable may require two, three, or more adjacent words. In either case, however, each variable starts in a new word in physical memory. Details of these three different storage and / or array formats are given in the context of the diagrams described below.

[0041] The term "feature map" can refer to one layer of a three-dimensional matrix, such as that shown in the upper left portion of Figure 2. Thus, one location in the plane represents one pixel throughout the depth of the plane.

[0042] A detailed description of the drawings is provided below. All descriptions in the drawings are schematic. First, a block diagram of one embodiment of a memory controller circuit of the present invention for mapping convolutional neural network data to physical memory is provided. Thereafter, further embodiments are described, as well as one embodiment of a method for mapping convolutional neural network data to physical memory.

[0043] FIG. 1 shows a block diagram of a preferred embodiment of a memory controller circuit 100 for mapping data of a convolutional neural network to a physical memory 106. The memory controller circuit 100 includes a receiving unit 102 for receiving a selection parameter value and a mapping unit 104 for mapping pixel values ​​of one layer of the convolutional neural network, i.e., output values ​​of an activation function of one layer of the CNN, to memory words of the physical memory 106 according to one of a plurality of mapping schemes, the mapping being performed depending on the received selection parameter value. As mentioned above, the transition or switching from one mapping scheme to another is bound to a specific application setup, but can also be performed dynamically and independently of the specific application setup. This makes the proposed concept flexible for any application field.

[0044] Furthermore, note that the physical memory 106 is shown in dashed lines since the proposed concept of memory controller circuitry can function without physical memory.

[0045] 2 shows a block diagram of one embodiment 200 of operations within a CNN, including a feature map 202, a (convolution) kernel matrix 208, and an associated output 204 in the form of a matrix 204, e.g., a feature map. Generally, in a convolution operation, a kernel or kernel matrix 208 is moved cycle by cycle over a larger feature map matrix 202 to generate one output value 210 of the result matrix 204. Due to the large number of matrix / vector operations / multiplications and the respective read and write operations from word-oriented physical memory, the form of mapping of individual values ​​of matrix elements can significantly impact the overall performance of the associated computations.

[0046] The bottom portion of Figure 2 shows that kernel matrix 208 is shifted downward by one element for the next cycle, producing a second result value 212 in the associated output matrix 204. Note further that in the top portion of Figure 2, feature map 202 is shown as one of a set of feature maps 206. This is generally true for the bottom portion of Figure 2 as well, but is not explicitly shown here.

[0047] FIG. 3 shows a block diagram of an idealized embodiment 300 of the concept proposed herein. Here, each element of the kernel matrix 208 is shown as a different pattern. However, the different element squares can also be associated with the exemplary nine elements of the feature map 202. Each pattern can be associated with a "pixel" or a "pixel value" (both terms are used equivalently in this description, and those skilled in the art would know how to distinguish between them). It is clear that numerous physical memory read and write operations may be required, especially when the size (i.e., one dimension of the matrix 202) and each set 206 contains hundreds or even thousands of elements.

[0048] In an ideal case, the values ​​of the matrix elements 202 represented for the matrix 208 can be mapped to the physical memory 302 represented using word boundaries. Each square with a different pattern represents one word with the same word size 304 measured in bits. However, it is not typical for pixel values ​​(i.e., the storage representation of a matrix element) to require exactly the same amount of bits as are available in a word of physical memory. A word of physical memory may contain 64 bits, while matrix elements of the feature map 206 may be 8, 16, 24, or more bits for computational efficiency. Therefore, the storage requirements of pixel values ​​and word size generally do not match perfectly.

[0049] FIG. 4 shows a block diagram of one embodiment of an IWAP scheme for mapping pixel values ​​(or matrix elements) of an exemplary matrix 402 to physical memory 404. Here, two memory words, each having a word size 406, are shown side-by-side as a vertically-spaced array. In an intra-word contiguous mapping scheme, pixel values ​​are stored adjacent to each other in memory, so that different pixel values ​​(or portions thereof) may be stored in the same physical memory word. The placement or mapping resembles a logical structure of data, where adjacent pixel values ​​in the same row are stored adjacent to each other. Each pixel value is thereby mapped so that different channels can be stored one after the other according to the logical size of the feature map matrix. This then assumes that a pixel is a vector of data, and thus a channel is an element of the vector, with each element, or channel, of the vector having a value represented in bits.

[0050] Now, if space is available within a word, different pixel values ​​can be stored in the same word. This is the case, for example, for pixel (0,0) and pixel (0,1) in word 1 (i.e., the top left corner). To cache a given number of pixel values, this method or scheme requires a minimal amount of memory. However, it may require bit indexing for both storing and fetching pixel values, and depending on the timestamp (cycle), may require different numbers of cycles to write and read pixel values.

[0051] FIG. 5 shows a block diagram of one embodiment 500 of a KLIP scheme for mapping pixels, or more appropriately, pixel values, to physical memory 404. Using this method, different pixel values ​​are stored in the same word only if they belong to a single row of the input patch for which a vector / matrix multiplication must be computed. Furthermore, here (as in the previous example), the feature map volume / matrix is ​​convolved with a 3x3 kernel. Highlighted in the figure are the rows of pixel values ​​belonging to the input patch for convolution: ((0,0), (0,1), (0,2)); ((1,0), (1,1), (1,2)); ((2,0), (2,1), (2,2)). The pixel values ​​of each row are stored consecutively in physical memory words. However, the remaining bit lines in the word containing the last pixel of the feature map value remain empty. This practice of leaving parts of memory empty to allow mapping of data closer to its logical arrangement is somewhat similar to the concept of memory interleaving commonly used in memory management for CPUs or other processing units (e.g., GPUs).

[0052] It is clear that this mapping scheme does not use the minimum amount of memory possible for one volume of pixel values, leaving some memory words empty to accommodate the positioning or placement of pixel values ​​in memory. Similar to the previous mapping scheme (i.e., IWAP), writing pixel values ​​may require different numbers of cycles depending on the time step (i.e., contention step). Furthermore, assuming different pixel values ​​can still be stored within one physical memory word, a bit-line index is required for writing. As mentioned previously, at every time step (i.e., every computation cycle), a new row of the input patch must be loaded into memory if a computation is to be performed. Because pixel values ​​are mapped contiguously, the number of read cycles is constant at every time step, and no bit-line index is required to read one complete row.

[0053] 5, it can be clearly seen that the memory word with index (1,0), i.e., the right-hand memory word, contains or stores a first portion 502 of pixel values ​​of the third horizontal element 504 of the exemplary matrix 402. A second portion 506 of pixel values ​​of the exemplary horizontal element 504 of the exemplary matrix 402 is stored in the memory word with index (0,1), i.e., the left-hand memory word in the second row. The remaining portion of the memory word 506 with index (0,1) remains empty.

[0054] Thus, each new row of matrix 402 begins at the beginning of the next available word start in physical memory. As a result, typically the last word on a horizontal row of matrix 402 contains unused bits.

[0055] FIG. 6 shows a block diagram of one embodiment 600 of a PLIP scheme for mapping pixel values ​​to physical memory 404. Using this mapping scheme, bits from different pixel values ​​cannot be stored in the same word. Each pixel value is stored in a single physical memory word. The unoccupied portion of the word storing each pixel value is left empty. This is somewhat similar to the concept of kernel-level interleaving or mapping, but this time it applies to a single pixel value across the channel depth. However, if a pixel value, measured in bits, is larger than one physical memory word, the pixel value is moved across the word boundary to the next adjacent word, leaving the unoccupied portion of the second word empty. This scheme also works when the memory space required for a pixel value is larger than two, three, or more physical memory words. Therefore, it can be said to be universal.

[0056] This mapping scheme provides the finest granularity of stored logical units and therefore the most efficient use of local memory. Because every pixel value is individually interleaved, no bit-line index is required for writing stored words. In contrast to the previous mapping scheme, the number of both read and write cycles remains constant regardless of the pixel value being read or stored.

[0057] Arrows 602, 604, 606 illustrate, by way of example, the mapping of some of the first rows of matrix 402 onto two words of physical memory 404.

[0058] 7 illustrates a block diagram of one embodiment of a method 700 of the present invention for mapping data of a convolutional neural network to physical memory, including receiving (702) selected parameter values ​​and mapping (704) pixel values ​​of one layer of the convolutional neural network to memory words of physical memory according to one of a plurality of mapping schemes, the mapping occurring corresponding to the received selected parameter values.

[0059] Furthermore, it is expressly contemplated that the concepts proposed herein may be implemented entirely in hardware, entirely in software, or a combination thereof, including microcode.

[0060] Embodiments of the present invention can be implemented using virtually any type of computer, regardless of the platform suitable for storing and / or executing program code. Figure 8 shows, by way of example, a computing system 800 suitable for executing program code associated with the proposed method 700.

[0061] Computing system 800 is merely one example of a suitable computer system, and whether computer system 800 is capable of implementing and / or performing any of the above-identified functions is not intended to suggest any limitation on the scope of use or functionality of the embodiments of the invention described herein. Computer system 800 has components operable with many general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 800 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices. Computer system / server 800 may be described in the general context of computer system-executable instructions, such as program modules, executed by computer system 800. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system / server 800 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0062] As shown, computer system / server 800 is depicted in the form of a general-purpose computing device. Components of computer system / server 800 include, but are not limited to, one or more processors or processing units 802, system memory 804, and a bus 806 that couples various system components, including system memory 804, to processor 802. Bus 806 may represent any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, but not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus. Computer system / server 800 typically includes a variety of computer system-readable media. Such media can be any available media that is accessible by computer system / server 800 and includes both volatile and non-volatile media, removable and non-removable media.

[0063] The system memory 804 may include computer-system-readable media in the form of volatile memory, such as random access memory (RAM) 808 or cache memory 810, or both. The computer system / server 800 may also include other removable / non-removable, volatile / non-volatile computing system storage media. By way of example only, a storage system 812 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, but typically referred to as a "hard drive"). Also included may be a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 806 by one or more data media interfaces. As further illustrated and described below, memory 804 may include at least one program product having a set (e.g., at least one) of program modules configured to implement the functionality of embodiments of the present invention.

[0064] A program / utility having a set (at least one) of program modules 816 can be stored in memory 804 along with, for example, but not limited to, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, can include a network environment implementation. The program modules 816 generally perform the functions and / or methods of embodiments of the present invention described herein.

[0065] The computer system / server 800 may further communicate with one or more external devices 818, such as a keyboard, pointing device, display 820; one or more devices that allow a user to interact with the computer system / server 800; or any device (e.g., a network card, modem, etc.) that allows the computer system / server 800 to communicate with one or more other computing devices. Such communication may occur via an input / output (I / O) interface 814. Additionally, the computer system / server 800 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 822. As shown, the network adapter 822 may communicate with other components of the computer system / server 800 via a bus 806. It should be understood that other hardware and / or software components, not shown, may be used in connection with the computer system / server 800. Examples include, but are not limited to, microcode, device drives, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0066] Furthermore, a memory mapping circuit 100 can be attached to the bus system 806 for mapping data of the convolutional neural network into a physical memory 808. To this end, the memory mapping circuit 100 can also comprise a bus interface circuit.

[0067] The descriptions of various embodiments of the present invention are presented for illustrative purposes, but they are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0068] The present invention may be a system, a method, or a computer program product, or a combination thereof, and may include a computer-readable storage medium or media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0069] The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system for a propagation medium. Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read-only memory (CD-ROM), compact disk-read / write (CD-R / W), DVD, and Blu-Ray disk.

[0070] A computer-readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. As used herein, computer-readable storage media is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through fiber optic cable), or electrical signals sent through wires.

[0071] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper cables, optical fibers, wireless networks, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0072] The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information in the computer readable program instructions to implement aspects of the present invention.

[0073] Aspects of the present invention will be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0074] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts or block diagrams, or both. These computer program instructions, which can direct a computer, other programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, can also be stored in a computer-readable medium, whereby the instructions stored in the computer-readable medium include an article of manufacture containing instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts or block diagrams, or both.

[0075] The computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer or other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowchart or block diagram, or both.

[0076] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or a combination of dedicated hardware and computer instructions.

[0077] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, it will be understood that the terms "comprise" and / or "comprising," when used herein, specify the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups, or combinations thereof.

[0078] Corresponding structure, materials, acts, and equivalents of "means-or-step-plus-function" elements in the following claims are intended to include any structure, material, or acts for performing the functions in conjunction with other claimed elements as specifically claimed. The description of the present invention has been presented only for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The embodiments have been chosen and described to best explain the principles and practical applications of the invention and to enable others skilled in the art to understand the invention in various embodiments with various modifications as suited to the particular uses contemplated.

Claims

1. 1. A memory controller circuit for mapping data of a convolutional neural network to physical memory, comprising: a receiving unit for receiving a selected parameter value; a mapping unit for mapping pixel values ​​of a layer of the convolutional neural network to memory words of the physical memory according to one of a plurality of mapping schemes, the mapping corresponding to the values ​​of the received selected parameter values; Equipped with said one of said plurality of mapping schemes is an intra-word contiguous mapping scheme, in which bit values ​​of portions of different pixel values ​​are mapped contiguously into one physical memory word; A memory controller circuit.

2. 2. The memory controller circuit of claim 1, wherein said one of said plurality of mapping schemes is a kernel word interleaving scheme, and bit values ​​of portions of different pixel values ​​are mapped into said physical memory such that at most a number of pixels equal to a kernel size of a convolutional neural network operation are contiguously mapped into adjacent memory words.

3. 2. The memory controller circuit of claim 1, wherein said one of said plurality of mapping schemes is an activation word interleaving scheme, and bit values ​​of pixel values ​​are mapped such that a memory word contains the bit values ​​of at most one pixel.

4. 2. The memory controller circuit of claim 1, wherein the selected parameter value is included in a set of parameter values ​​for a layer of the convolutional neural network.

5. 2. The memory controller circuit of claim 1, wherein the selected parameter values ​​are independent of parameter values ​​of layers of the convolutional neural network.

6. 2. The memory controller circuit of claim 1, wherein the physical memory is selected from the group consisting of static random access memory, latch-based memory, and dynamic random access memory.

7. 10. The memory controller circuit of claim 1, wherein the memory controller circuit is integrated with physical memory.

8. 2. The memory controller circuit of claim 1, wherein the physical memory stores digital memory input values ​​and is integrated with a digital processing unit.

9. 1. A method for mapping data of a convolutional neural network to physical memory, comprising: receiving a selection parameter value; mapping pixel values ​​of a layer of the convolutional neural network to memory words of the physical memory according to one of a plurality of mapping schemes, the mapping corresponding to the values ​​of the received selected parameter values; Including, said one of said plurality of mapping schemes is an intra-word contiguous mapping scheme, in which bit values ​​of portions of different pixel values ​​are mapped contiguously into one physical memory word; method.

10. 10. The method of claim 9, wherein said one of said plurality of mapping schemes is a kernel word interleaving scheme, and bit values ​​of portions of different pixel values ​​are mapped into said physical memory such that at most a number of pixels equal to a kernel size of a convolutional neural network operation are contiguously mapped into adjacent memory words.

11. 10. The method of claim 9, wherein said one of said plurality of mapping schemes is an activation word interleaving scheme, and bit values ​​of pixel values ​​are mapped such that a memory word contains the bit values ​​of at most one pixel.

12. 10. The method of claim 9, wherein the selected parameter values ​​are included in a set of parameter values ​​for a layer of the convolutional neural network.

13. 10. The method of claim 9, wherein the selected parameter values ​​are independent of parameter values ​​of layers of the convolutional neural network.

14. 10. The method of claim 9, wherein the physical memory is selected from the group consisting of static random access memory, latch-based memory, and dynamic random access memory.

15. The method of claim 9, wherein the mapping is performed by a memory controller circuit integrated with the physical memory.

16. A computer program comprising computer readable program instructions for causing a computer to carry out the method of any one of claims 9 to 15.

17. 17. A computer readable storage medium having stored thereon the computer program of claim 16.

Images