Data processing systems, their operating methods, and the computing systems that use them.

Abstract

This disclosure relates to a data processing system comprising: a controller configured to receive a neural network operation processing request from a host device; and an internal memory computing device including a plurality of processing elements. The internal memory computing device is configured to receive an input feature map and a weight filter from the controller, and to perform a neural network operation in the plurality of processing elements based on the weight filter and a plurality of partition maps generated from the input feature map. The internal memory computing device performs the neural network operation by not moving reused elements between processing elements; the reused elements are elements constituting the partition maps that are computed at least twice during the neural network operation.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2021-0116331, filed on September 1, 2021, which is incorporated herein by reference in its entirety. Technical Field

[0003] The various embodiments of this disclosure generally relate to a data processing technique, and more specifically, to a data processing system for neural network operations, a method of operating the system, and a computing system using the data processing system. Background Technology

[0004] As people's interest in and emphasis on artificial intelligence applications and big data analysis increase, the demand for computing systems capable of efficiently processing massive amounts of data is also growing.

[0005] Artificial neural networks are one way to implement artificial intelligence. The goal of artificial neural networks is to improve a machine's problem-solving abilities, specifically by enhancing its reasoning capabilities through learning. However, as output accuracy increases, so too may computational complexity, memory access frequency, and data movement.

[0006] This could lead to reduced speed, increased power consumption, and thus degraded system performance. Summary of the Invention

[0007] In embodiments of this disclosure, a data processing system may include: a controller configured to receive a neural network operation processing request from a host device; and an internal memory computing device including a plurality of processing elements. The internal memory computing device is configured to receive an input feature map and a weight filter from the controller, and to perform a neural network operation in the plurality of processing elements based on the weight filter and a plurality of partition maps generated from the input feature map. The internal memory computing device performs the neural network operation by not moving reused elements between processing elements; the reused elements are elements constituting the partition maps that are computed at least twice during the neural network operation.

[0008] In embodiments of this disclosure, a data processing system may include: a global buffer storing an input feature map and a weight filter; a computation memory including a plurality of processing elements and configured to perform multiple loops of neural network operations by receiving the weight filter and a plurality of partition maps generated from the input feature map; and a scheduler configured to: select a processing element among the plurality of processing elements corresponding to the number of elements of the weight filter, store all elements of the weight filter in the selected processing element, and allocate and store elements of each partition map in the selected processing element, wherein the scheduler allocates and stores elements of each partition map by allowing reused elements to remain in the respective single processing element among the plurality of processing elements where the reused element is first provided, the reused element being an element among the elements of the partition map that is computed at least twice during the neural network operation.

[0009] In embodiments of this disclosure, a method of operating a data processing system may include: receiving a neural network operation processing request from a host device by a controller; receiving an input feature map and a weight filter from the controller by a built-in memory computing device including multiple processing elements; generating multiple partition maps by the built-in memory computing device based on the input feature map; and performing a neural network operation by the built-in memory computing device based on the multiple partition maps and the weight filter, through at least some of the multiple processing elements, wherein performing the neural network operation includes controlling a reused element not to move between processing elements, the reused element being an element constituting the partition map that is computed at least twice during the neural network operation.

[0010] In embodiments of this disclosure, a computing system may include: a host device; and a data processing system. The data processing system is configured to, in response to a neural network operation processing request from the host device, generate multiple partition maps based on an input feature map, and perform neural network operations in multiple processing elements based on weighted filters and the multiple partition maps. The data processing system performs the neural network operations by not moving reused elements between processing elements; the reused elements are elements constituting the partition maps that are computed at least twice during the neural network operation.

[0011] In embodiments of this disclosure, a built-in memory computing device includes: a processing element (PE) configured to perform a convolution operation on a filter and a partition map in each loop, each PE being configured to perform a convolution operation on an assigned filter element and an assigned map element; and a control unit configured to: assign filter elements from the filter to each PE, partition an input feature map into a partition map such that some map elements are shared by two in the partition map, and in each loop, assign map elements from a selected partition map to each PE, wherein the control unit is further configured to control the selected PE to perform a convolution operation on a recirculated map element in the current loop without reassigning the recirculated map element to the selected PE, and wherein the control unit assigned the recirculated map element to the selected PE in the previous loop.

[0012] These and other features, aspects, and embodiments are described in more detail below. Attached Figure Description

[0013] The above and other aspects, features, and advantages of the subject matter of this disclosure will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, wherein:

[0014] Figure 1 This is a diagram illustrating the configuration of a computing system according to an embodiment of the present disclosure;

[0015] Figure 2 This is a diagram used to illustrate the data processing concept of an artificial neural network according to embodiments of the present disclosure;

[0016] Figure 3 This is a diagram used to illustrate the operational concepts of convolutional layers according to embodiments of the present disclosure;

[0017] Figure 4 This is a diagram illustrating the configuration of a neural network processor according to an embodiment of the present disclosure;

[0018] Figure 5 This is a diagram illustrating the configuration of a scheduler according to an embodiment of the present disclosure;

[0019] Figure 6 This is a diagram illustrating the configuration of a computing memory according to an embodiment of the present disclosure;

[0020] Figures 7 to 10 These are diagrams illustrating the concept of data reuse according to embodiments of the present disclosure; and

[0021] Figure 11A and Figure 11B This is a graph used to describe the data processing efficiency of the data reuse method according to embodiments of the present disclosure. Detailed Implementation

[0022] Embodiments of this disclosure are described in detail with reference to the accompanying drawings. The drawings are schematic diagrams of various embodiments (and intermediate structures). Therefore, variations in the illustrated configurations and shapes are expected due to factors such as manufacturing techniques and / or tolerances. Consequently, the described embodiments should not be construed as limited to the specific configurations and shapes shown herein, but may include deviations in configuration and shape without departing from the spirit and scope of this disclosure as defined by the appended claims.

[0023] This document describes embodiments with reference to cross-sectional and / or plan views of various embodiments of the present disclosure. However, the embodiments of the present disclosure should not be construed as limiting the present disclosure. Although several embodiments of the present disclosure have been shown and described, those skilled in the art will understand that changes can be made to these embodiments without departing from the principles and spirit of the present disclosure.

[0024] Figure 1 This is a diagram illustrating a computing system according to an embodiment of the present disclosure.

[0025] The computing system 10 may include a host device 100 and a data processing system 200 configured to perform operational processing on an application requested by the host device 100.

[0026] The host device 100 may include at least one intellectual property (IP) block, such as a main processor 110, random access memory (RAM) 120, memory 130, input / output (I / O) devices (140), etc. The host device may further include general-purpose components (not shown).

[0027] In this embodiment, the host device 100 may be implemented as a system-on-a-chip (SoC), wherein the components of the host device 100 are integrated into a single semiconductor chip, but is not limited thereto. The components of the host device 100 may be implemented using multiple semiconductor chips.

[0028] The main processor 110 can control all operations of the computing system 10, and the main processor 110 can be, for example, a central processing unit (CPU). The main processor 110 may include one or more cores. The main processor 110 can process or run programs, data, and / or instructions stored in RAM 120 and memory 130. For example, the main processor 110 can control the functions of the computing system 10 by running programs stored in memory 130.

[0029] RAM 120 can temporarily store programs, data, or commands. Programs and / or data stored in memory 130 can be temporarily loaded into RAM 120 under the control of the main processor 110 or according to boot code. RAM 120 can be implemented using memory such as dynamic RAM (DRAM) or static RAM (SRAM).

[0030] Memory 130 can be a storage site for storing data, and for example, memory 130 can store an operating system (OS) as well as various types of programs and data. Memory 130 can include at least one of volatile memory and non-volatile memory. Non-volatile memory can be selected from read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), and ferroelectric RAM (FRAM). Volatile memory can be selected from dynamic RAM (DRAM), static RAM (SRAM), and synchronous DRAM (SDRAM). In embodiments, memory 130 can be implemented as a hard disk drive (HDD), solid-state drive (SSD), compact flash (CF) card, secure digital card (SD) card, micro secure digital card (microSD) card, mini secure digital card (miniSD) card, extreme digital card (xD) card, or memory stick.

[0031] I / O device 140 can receive user input or input data from the outside and output the data processing results of computing system 10. I / O device 140 can be implemented as a touch screen panel, keyboard, various types of sensors, etc. In embodiments, I / O device 140 can collect environmental information of computing system 10. For example, I / O device 140 may include imaging device and image sensor, and I / O device 140 can sense or receive image signals from outside data processing system 200, convert the sensed or received image signals into image data, and store the image data in memory 130 or provide the image data to data processing system 200.

[0032] In response to a request from the host device 100, the data processing system 200 can extract valid information by analyzing input data based on an artificial neural network, and determine the status or control elements of the electronic device on which the data processing system 200 is installed based on the extracted information. For example, the data processing system 200 can be applied to drones, advanced driver assistance systems (ADAS), smart TVs (TVs), smartphones, medical devices, mobile devices, image display devices, measuring devices, Internet of Things (IoT) devices, etc. Furthermore, the data processing system 200 can be installed on one of various types of computing systems 10.

[0033] In an embodiment, the host device 100 can offload neural network operations to the data processing system 200, and the host device 100 can provide the data processing system 200 with initial parameters for neural network operations, such as input data and weights.

[0034] In one embodiment, the data processing system 200 may be an application processor installed on a mobile device.

[0035] The data processing system 200 may include at least a neural network processor 300.

[0036] The neural network processor 300 can generate neural network models by training or learning from input data, generate information signals by performing calculations on the input data based on the neural network model, or retrain the neural network model. Neural networks can include, but are not limited to, various types of neural network models, such as convolutional neural networks (CNNs), region convolutional neural networks (R-CNNs), region proposal networks (RPNs), recurrent neural networks (RNNs), stacked deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNNs), deconvolutional networks, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory (LSTM) networks, and classification networks.

[0037] Figure 2 This is a diagram used to illustrate the data processing concept of an artificial neural network according to embodiments of the present disclosure. Figure 2 This illustrates the data processing concept of CNN.

[0038] CNNs can be constructed from convolutional layers, pooling layers, and fully connected layers.

[0039] Convolutional layers can generate output feature maps OFM by applying weighted filters (kernels) W to the input feature map IFM.

[0040] Pooling layers can be used to add spatial invariance to features extracted by convolutional layers and can also be used to reduce the output of convolutional layers.

[0041] Convolutional and pooling layers can significantly reduce the complexity of the entire model by reducing the number of parameters in the neural network.

[0042] Fully connected layers can generate output data by classifying input data based on feature extraction results from pooling layers.

[0043] Figure 3 This is a diagram used to illustrate the operational concept of a convolutional layer according to embodiments of the present disclosure.

[0044] The input feature map IFM and weight filter W can be provided in matrix form. Unless otherwise specified below, it should be understood that the input feature map IFM and weight filter W can be matrices with a set dimension (rows * columns) or a set size.

[0045] To apply the weight filter W to the input feature map IFM, the input feature map IFM can be partitioned into multiple partition maps IDIV11, IDIV12, IDIV13, and IDIV14, each with the same dimensions as the weight filter W. For example, the input feature map IFM can be partitioned into multiple partition maps IDIV11, IDIV12, IDIV13, and IDIV14 when a convolutional window with the same dimensions as the weight filter W slides at fixed intervals (i.e., strides) using reference elements REF located in the first row and first column of the input feature map IFM, and the weight filter W can be applied sequentially to partition maps IDIV11, IDIV12, IDIV13, and IDIV14. Therefore, the output feature map OFM can be computed by performing a multiplication and accumulation (MAC) operation on the elements of partition maps IDIV11, IDIV12, IDIV13, and IDIV14 and the elements of the weight filter W.

[0046] Figure 3 The input feature map IFM is shown to be I*I (where I = 5), the weight filter W is K*K (where K = 3), and the stride is 2. The number of partition maps IDIV11, IDIV12, IDIV13, and IDIV14 can be determined as (IK)*(IK). To perform convolution operations on the input feature map IFM, an output feature map OFM of size O*O (where O = IK) can be obtained by performing convolution operations on the weight filter W and each partition map IDIV11, IDIV12, IDIV13, and IDIV14 via convolution loops. Therefore, the number of convolution loops can be the same as the number of partition maps IDIV11, IDIV12, IDIV13, and IDIV14.

[0047] The convolution operation can be performed by sliding the convolution window along the row or column direction with a stride, starting with the first convolution loop of applying the weight filter W to the first partition map IDIV11, which includes the reference element REF. For example, when performing a convolution operation on the partition map IDIV12, which includes the last element A15 in the row, by sliding the convolution window along the row direction, the convolution window can be slid along the column direction. Then, by repeating the process of performing convolution operations by sliding the convolution window again along the row direction, convolution operations can be performed on all partition maps IDIV11, IDIV12, IDIV13, and IDIV14 with the weight filter W, thereby obtaining the output feature map OFM.

[0048] from Figure 3It can be seen that when the weight filter W is applied to each of the partition maps IDIV11, IDIV12, IDIV13 and IDIV14 of the input feature map IFM, the weight filter W is reused in each convolution loop, and at least some elements of the partition maps IDIV11, IDIV12, IDIV13 and IDIV14 are reused.

[0049] For example, when convolution is performed on the first partition map IDIV11 and the weight filter W in the first convolution loop, and then on the second partition map IDIV12 and the weight filter W in the second convolution loop, the elements A13, A23 and A33 used in the first convolution loop can be reused.

[0050] When performing convolution operations on the third partition map IDIV13 and the weighted filter W in the third convolution loop, the elements A31, A32, and A33 used in the first convolution loop and the element A33 used in the second convolution loop can be reused.

[0051] When performing convolution operations on the fourth partition map IDIV14 and the weighted filter W in the fourth convolution loop, the elements A33, A34, and A35 used in the second convolution loop and the elements A33, A43, and A53 used in the third convolution loop can be reused.

[0052] The data used in the convolution operation, such as the weighted filter W and the input feature map IFM, can be provided to the processing element PE from an external buffer. The external buffer and the processing element PE will be described in detail later.

[0053] In this embodiment, the convolution operation on the elements of the partition graphs IDIV11, IDIV12, IDIV13, and IDIV14 can be processed in independent processing elements PE. However, in this embodiment, it is difficult to guarantee power consumption and latency characteristics when elements used in the previous convolution loop that will be reused in the next convolution loop are again input from the external buffer to the processing element PE or moved between processing elements PE.

[0054] Therefore, this disclosure provides a method for minimizing the amount of data movement used repeatedly in a convolution loop.

[0055] Figure 4 This is a diagram illustrating the configuration of a neural network processor according to an embodiment of the present disclosure.

[0056] Reference Figure 4The neural network processor 300 may be a dedicated processor or accelerator for neural network operations, and may include a built-in memory computing device 310, a controller 320, and RAM 330. In embodiments, the neural network processor 300 may be implemented as a system-on-a-chip (SoC) integrated into a single semiconductor chip, but is not limited thereto. The neural network processor 300 may be implemented using multiple semiconductor chips.

[0057] The controller 320 can control all operations of the neural network processor 300. The controller 320 can set and manage parameters related to neural network operations to allow the built-in memory computing device 310 to operate the neural network operations correctly. The controller 320 can be implemented using hardware, software (or firmware), or a combination of hardware and software.

[0058] The controller 320 can be implemented using at least one processor, such as a central processing unit (CPU), microprocessor, etc., and can execute instructions stored in RAM 330 to perform various functions.

[0059] RAM 330 can be implemented using DRAM, SRAM, etc., and RAM 330 can store various types of programs and data for the operation of controller 320, as well as data generated in controller 320.

[0060] The built-in memory computing device 310 can be configured to perform neural network operations under the control of the controller 320. The built-in memory computing device 310 may include a computing memory 311, a global buffer 313, an accumulator (ACCU) 315, an activator (ACTIV) 317, a pooler (POOL) 319, and a scheduler 500.

[0061] The computation memory 311 may include multiple processing elements PE. Each of the processing elements PE can perform a convolution operation on elements of the input feature map IFM provided from the global buffer 313 and elements of the weight filter W. For example, each of the processing elements PE can perform a multiplication and accumulation (MAC) operation on elements of the partition map in the input feature map IFM and elements of the weight filter W.

[0062] The global buffer 313 can store the input feature map IFM and the weighted filter W, and provide the stored input feature map IFM and the stored filter W to the computation memory 311. The global buffer 313 can receive the operation results from the computation memory 311 and store the received operation results therein. The global buffer 313 can be implemented using DRAM, SRAM, etc.

[0063] Accumulator 315 can be configured to derive a weighted sum by accumulating the processing results of processing element PE.

[0064] Activator 317 can be configured to add nonlinearity by applying an activation function such as a rectified linear unit (ReLU) to the weighted sum result of accumulator 315.

[0065] Pooler 315 can reduce and optimize dimensionality by sampling the output value of activator 317.

[0066] The processing performed by the computation memory 311, accumulator 315, activator 317 and pooler 319 can be a process of learning or relearning a neural network model or a process of inferring input data.

[0067] During the learning of neural network models or inference of input data, data movement from the global buffer 313 to the computation memory 311 and between processing elements PE in the computation memory 311 may lead to increased power consumption and reduced data processing speed.

[0068] According to this disclosure, the scheduler 500 can be configured to schedule the transfer methods of the weighted filter W and the input feature map IFM to be used in the processing element PE, so as to minimize data movement during the data processing of the computing memory 311.

[0069] Figure 5 This is a diagram illustrating a scheduler according to an embodiment of the present disclosure.

[0070] Reference Figure 5 The scheduler 500 according to the embodiment may include a weighted filter provider 510, a partition graph configurator 520, and a movement path controller 530.

[0071] In order to perform convolution operations on each element of the partition graph IDIV and the element of the weight filter W simultaneously and independently in each unit convolution loop, the number of processing elements PE may be the same as the number of elements of the partition graph IDIV or the same as the number of elements of the weight filter W.

[0072] The weight filter provider 510 can select processing elements PE according to the number required in each convolution loop and provide all elements of the weight filter W to the selected processing elements PE. Therefore, all elements of the weight filter W can be stored in the selected processing elements PE.

[0073] The partition map configurator 520 can partition the input feature map IFM into multiple partition map IDIVs, each of which has the same size as the weight filter W. In an embodiment, the partition map configurator 520 can obtain multiple partition map IDIVs by sliding a convolutional window of the same size as the weight filter W at fixed intervals (stride) using reference elements located in the first row and first column of the input feature map IFM. When the input feature map IFM is partitioned into multiple partition map IDIVs, convolution operations can be performed in convolutional loops, the number of loops being the same as the number of partition map IDIVs.

[0074] The movement path controller 530 can be configured to provide elements of the partition map IDIV to the computation memory 311 during sequential execution of convolution loops. The elements of the partition map IDIV used in each convolution loop can be distributed and provided non-overlapping to multiple processing elements PE selected by the weight filter provider 510.

[0075] Specifically, the movement path controller 530 can be configured to provide new elements that are to be used for the first time in the compute memory 311, such as elements that have no transfer history from the global buffer 313 to the compute memory 311, to the compute memory 311.

[0076] Furthermore, the movement path controller 530 can control reusable elements to be reused in the compute memory 311, such as elements with a transfer history from the global buffer 313 to the compute memory 311, so that they are not transferred from the global buffer 313 to the compute memory 311. It can also control reusable elements not to be transferred between processing elements PE in the compute memory 311. Therefore, reusable elements can be processed in the following state: the reusable element is retained in the processing element where it was previously used.

[0077] The elements of a partition map IDIV to be used in a specific convolution loop can include only new elements or only reused elements. Alternatively, the elements of a partition map IDIV to be used in a specific convolution loop can include both new and reused elements.

[0078] According to the present disclosure, the movement path controller 530 can determine the reuse state (reuse / non-reuse and reuse cycle) of each element in the partition graph IDIV during the convolution loop. Based on the determination result, the movement path controller 530 can control the reuse of elements so that the reused element is computed in the processing element PE that computed the reused element in the previous loop or in the processing element PE that computed the reused element for the first time, thereby minimizing the amount of movement of the reused element.

[0079] In an embodiment, the movement path controller 530 can control the movement path of elements to be reused when the convolutional window slides along the row direction and / or when the convolutional window slides along the column direction. The reused elements are overlapping elements between multiple partition maps IDIVs generated by moving the input feature map along the row and / or column directions of the convolutional window, and have already been computed with the weight filter W in the previous convolutional loop.

[0080] For example, the move path controller 530 can control elements intended to be reused only in the row direction, ensuring they are retained in the processing elements PEs where these elements were used in the previous cycle, and then reused in the next cycle. Similarly, the move path controller 530 can control elements intended to be reused only in the column direction, ensuring they are retained in the processing elements PEs where these elements were used in the previous cycle, and then reused in the next cycle. In another example, the move path controller 530 can control all elements intended to be reused in both the row and column directions, ensuring they are reused in the processing elements PEs where these elements were used in the previous cycle in the next cycle.

[0081] Figure 6 This is a diagram illustrating the configuration of a computing memory according to an embodiment of the present disclosure.

[0082] Reference Figure 6 The computing memory 400 according to the embodiment may include a plurality of tiles.

[0083] Each of the multiple tiles may include a tile input buffer 410, multiple processing elements PE, and an accumulation and tile output buffer 420.

[0084] Each of the multiple processing elements PE may include a PE input buffer 430, multiple subarrays SA, and an accumulator PE output buffer 440.

[0085] The subarray SA can refer to a synaptic array and may include multiple word lines WL1, WL2, ..., WLN, multiple bit lines BL1, BL2, ..., BLM, and multiple memory cells MC. In an embodiment, the memory cell MC may include a resistive memory device RE, specifically a memristor device, but is not limited thereto. The value of the data to be stored in the memory cell MC can be changed by a write voltage applied via the multiple word lines WL1 to WLN or the multiple bit lines BL1 to BLM, and the resistive memory cell can store data by changing its resistance.

[0086] In embodiments, resistive memory cells may be implemented to include resistive memory cells such as PRAM cells, RRAM cells, MRAM cells, and FRAM cells.

[0087] The resistive device constituting the resistive memory cell may include, but is not limited to, materials whose crystalline state changes according to the amount of current, such as phase change materials, perovskite compounds, transition metal oxides, magnetic materials, ferromagnetic materials or antiferromagnetic materials.

[0088] When the unit cell of the subarray SA is a memristor device, the processing element PE can store the data corresponding to each element of the weighted filter W in the memristor device, and can perform convolution operations using Kirchhoff's laws or Ohm's laws by applying the voltage corresponding to the element of the partitioned graph IDIV to the word lines WL1 to WLN.

[0089] When the convolution window size is, for example, 2x2, four processing elements (PEs) may be needed in each convolution loop to perform convolution operations on the elements of the partitioned graph IDIV. The subarray SA included in the processing element PE can be activated at least in part based on the number of times the reused elements are reused and the size of the weight filter W.

[0090] For convolution operations, elements of the weight filter W can be stored in different subarrays SA of all selected processing elements PE. Elements of the partition map IDIV can be assigned and provided to the selected processing elements PE. For example, elements of the partition map IDIV to be operated on in a convolution loop can be stored in different processing elements PE. In this example, new elements of the partition map IDIV can be transferred from the global buffer 313 to the processing element PE, and convolution operations can be performed while reused elements are retained in the processing element PE that used the reused elements in the previous convolution loop. Therefore, new elements of a particular convolution loop can be stored in processing element PEs that do not store reused elements. Furthermore, each element of the partition map IDIV can be provided to the subarray SA of the selected processing element PE, which stores the weight filter W to be operated on with that element.

[0091] Therefore, when performing convolution operations, the processing element PE does not need to transfer the elements of the partition map IDIV generated based on the input data to another processing element PE, and the processing element PE does not need to receive the elements of the partition map IDIV that were previously input to the processing element PE from the global buffer 313.

[0092] Figure 6The computation memory 400 shown may be merely an example and may be any structure capable of processing elements of the partition graph IDIV and convolution operations of the weighted filter W without moving reusable elements of the partition graph IDIV between processing elements PE.

[0093] Figures 7 to 10 This is a diagram used to illustrate the concept of data reuse according to embodiments of the present disclosure.

[0094] Figure 7 The weighted filter W and the input feature map IFM according to an embodiment are shown.

[0095] from Figure 7 It can be seen that the size of the weight filter W is 2*2, and the size of the input feature map IFM is 8*8.

[0096] like Figure 7 As shown, when performing convolution operations on the weight filter W and the input feature map IFM with a step size of 1, the convolution operation can be completed through 49 convolution cycles for {(8-1)*(8-1)} partition maps IDIV.

[0097] Figure 8 It shows Figure 7 The input feature map IFM shown is a partial partition map IDIV0 to IDIV13.

[0098] Figure 8 The partition maps IDIV0 to IDIV15 are shown in the first convolution cycle T0 to the 16th convolution cycle T15, which are respectively in 49 convolution cycles.

[0099] Elements "2" and "10" from the first partition map IDIV0 used in the first convolution loop T0 can be reused in the second convolution loop T1, and elements "3" and "11" from the second partition map IDIV1 used in the second convolution loop T1 can be reused in the third convolution loop T2. As described above, at least some elements of the partition map (IDIV) can be reused as the convolution window moves along the row direction according to the stride.

[0100] Figure 9 This is a diagram illustrating the amount of data movement in a row-by-row reuse method, where elements to be reused in the row direction have been retained in the processing element PE and then reused in the next convolution loop. About Figure 9The field "time" indicates the convolution loop, the field "SAidx" indicates the identifier of the subarray SA, "PIBUF" indicates the PE input buffer, and the field "GBUF" indicates the global buffer. The numbers in the cells from the left, from column 3 to column 18, represent the elements that are reused in each convolution loop. The numbers in the cells from column 19 to column 22 represent the number of times the PE input buffer PIBUF is updated. The number in the cell of column 23 represents the total number of times the PE input buffer PIBUF of each PE1, PE2, PE3, and PE4 is updated. The number in the cell of column 24 represents the number of times the global buffer GBUF is read.

[0101] like Figure 6 As shown, each processing element PE can include multiple subarrays SA. Elements of the weight filter W can be allocated and stored in different subarrays SA of all processing elements PE1 to PE4 selected for the convolution operation.

[0102] Reference Figure 8 and Figure 9 Since the elements "1", "9", "2", and "10" of the first partition graph IDIV0 used in the first convolution loop T0 (i.e., time = 0) are new elements, the new elements "1", "9", "2", and "10" can be obtained from the global buffer GBUF (see reference). Figure 4 313 or Figure 6 (430) is stored in the first processing element PE1, the second processing element PE2, the third processing element PE3, and the fourth processing element PE4. New elements “1”, “9”, “2”, and “10” can be allocated and provided to the selected first to fourth processing elements PE1 to PE4 respectively, wherein each of the new elements “1”, “9”, “2”, and “10” can be stored in any of the subarrays SA of the corresponding processing element PE.

[0103] Since elements "3" and "11" are new elements among the elements "2", "10", "3" and "11" of the second partition graph IDIV1 used in the second convolution loop T1 (i.e., time = 1), the new elements "3" and "11" can be obtained from the global buffer GBUF (see reference). Figure 4 313 or Figure 6 The reused elements “2” and “10” are provided by the global buffer GBUF. Since reused elements “2” and “10” have already been stored in the third processing element PE3 and the fourth processing element PE4, reused elements “2” and “10” do not need to be provided from the global buffer GBUF or moved to other processing elements. New elements “3” and “11” can be stored in the first processing element PE1 and the second processing element PE2 to be allocated together with the reused elements.

[0104] Since elements "4" and "12" are new elements among the elements "3", "11", "4" and "12" of the third partition graph IDIV2 used in the third convolution loop T2 (i.e., time=2), the new elements "4" and "12" can be obtained from the global buffer GBUF (see reference). Figure 4 313 or Figure 6 The elements “3” and “11” are provided by the global buffer GBUF. Since the reused elements “3” and “11” have already been stored in the first processing element PE1 and the second processing element PE2, the reused elements “3” and “11” do not need to be provided from the global buffer GBUF or moved to other processing elements. The new elements “4” and “12” can be stored in the third processing element PE3 and the fourth processing element PE4.

[0105] After performing the seventh convolution cycle T6 (i.e., time = 6) through the above repetition process, the convolution window can move along the column direction with a stride, and then the eighth to fourteenth convolution cycles T7 to T13 can be performed on the eighth to fourteenth partition maps IDIV7 to IDIV13 when the convolution window moves along the row direction with a stride again.

[0106] During the 49th convolution loop, when the elements to be reused in the row direction are kept in the state of the processing element PE that already stores the reused elements, the convolution operation is performed from the global buffer GBUF (see reference). Figure 4 313 or Figure 6 The data transfer count of 430) and the update count of the PE input buffer "PIBUF" are 112 and 112, respectively.

[0107] In pairs with a step size of "1" Figure 7 In the convolution operation of the weighted filter W and the input feature map IFM shown, when the global buffer GBUF (see reference) is used in each convolution loop... Figure 4 313 or Figure 6 When reading a new element and transferring a reused element to another processing element (PE), the data transfer count from the global buffer GBUF and the PE input buffer (refer to 430) are recorded. Figure 4 313 or Figure 6 The update counts for (430) are 112 and 196 respectively.

[0108] Therefore, when reused elements are not moved as in this disclosure, the amount of data movement can be significantly reduced.

[0109] Figure 10This is a diagram illustrating the amount of data movement in a row-by-column reuse method according to embodiments of the present disclosure, wherein elements to be reused in the row and column directions have been retained in the processing element PE and then reused in the next convolution loop. About Figure 10 The field "time" indicates the convolution loop, the field "SAidx" indicates the identifier of the subarray SA, "PIBUF" indicates the PE input buffer, and the field "GBUF" indicates the global buffer. The numbers in the cells from the left, from column 3 to column 18, represent the elements that are reused in each convolution loop. The numbers in the cells from column 19 to column 22 represent the number of times the PE input buffer PIBUF is updated. The number in the cell of column 23 represents the total number of times the PE input buffer PIBUF of each PE1, PE2, PE3, and PE4 is updated. The number in the cell of column 24 represents the number of times the global buffer GBUF is read.

[0110] Reference Figure 8 and Figure 10 Since elements "17" and "18" in the eighth partition graph IDIV7, which is used in the eighth convolution loop T7 (i.e., time=7) where elements are reused in the column direction, are new elements, the new elements "17" and "18" can be obtained from the global buffer GBUF (see reference). Figure 4 313 or Figure 6 (430) is provided. Since the reused elements “9” and “10” in the column direction have already been stored in the second processing element PE2 and the fourth processing element PE4 in the first convolution loop T0 (i.e., time = 0), the new elements “17” and “18” in the eighth convolution loop T7 (i.e., time = 7) can be stored in the first processing element PE1 and the third processing element PE3. In addition, the second processing element PE2 and the fourth processing element PE4 must continuously store the reused elements “9” and “10” from the first convolution loop T0 (i.e., time = 0) until the eighth convolution loop T7 (i.e., time = 7), wherein the reused elements “9” and “10” are first input in the first convolution loop T0 and will be reused in the eighth convolution loop T7.

[0111] Since element "19" is a new element among the elements "10", "18", "11" and "19" of the ninth partition graph IDIV8 used in the ninth convolution loop T8 (i.e., time=8), the new element "19" can be obtained from the global buffer GBUF (see reference). Figure 4 313 or Figure 6430) is provided. Since the reused elements “10” and “11” to be reused in the column direction have been stored in the fourth processing element PE4 and the second processing element PE2 in the first convolution loop T0 (i.e., time = 0) and the second convolution loop T1 (i.e., time = 1), and the reused element “18” to be reused in the row direction has been stored in the third processing element PE3 in the eighth convolution loop T7 (i.e., time = 7), the new element “19” in the ninth convolution loop T8 can be stored in the first processing element PE1. Furthermore, the fourth processing element PE4 must continuously store the reused element "10" starting from the first convolution loop T0 (i.e., time = 0) when the reused element "10" is first input; the second processing element PE2 must continuously store the reused element "11" starting from the second convolution loop T1 (i.e., time = 1) when the reused element "11" is first input; and the third processing element PE3 must continuously store the reused element "18" starting from the eighth convolution loop T7 (i.e., time = 7) when the reused element "18" is first input.

[0112] The first processing element PE1 must continuously store and reuse element "17" until the fifteenth convolution cycle T14 (i.e., time = 14) in which element "17" included in the fifteenth partition diagram IDIV14 will be reused. The third processing element PE3 must continuously store and reuse element "18" until the sixteenth convolution cycle T15 (i.e., time = 15) in which element "18" included in the sixteenth partition diagram IDIV15 will be reused.

[0113] When elements that are repeatedly used in the row and column directions during 49 convolution loops are operated on in a single processing element PE, the data comes from the global buffer GBUF (see reference). Figure 4 313 or Figure 6 The data transfer count and update count in the PE input buffer PIBUF are 64 and 64 respectively, thus reducing the amount of data movement compared to the row-by-row reuse method.

[0114] Therefore, the movement of elements that are reused in the row or column direction can be restricted, thereby significantly reducing the amount of data movement.

[0115] Figure 11A and Figure 11B This is a diagram illustrating the data processing efficiency of a data reuse method according to embodiments of the present disclosure.

[0116] Figure 11AThe diagram illustrates the performance factor measurements for the methods Prior, row-by-row reuse method A, and row-by-column reuse method B, which relate to the movement of data between processing elements PE. The performance factor can be categorized into TOPS / W (billions of operations per second per watt), POWER (power consumption), and data processing latency.

[0117] Figure 11B It shows Figure 11A The measured values ​​shown are relative to the relevant method Prior.

[0118] from Figure 11B As can be seen, the performance factor of method A, which reuses rows, is improved compared to the related method Prior. Specifically, the latency is significantly improved by approximately 10%.

[0119] from Figure 11B It can be seen that, compared with the related methods Prior and the row-by-row reuse method A, the row-by-column reuse method B has significantly improved performance.

[0120] Because data movement is restricted in the neural network processor configured to process massive amounts of data as described in this disclosure, speed reduction and bottlenecks due to data transfer overhead can be prevented. Furthermore, power consumption can be significantly reduced and data processing speed can be increased, thereby enabling efficient neural network operations.

[0121] The embodiments described above are intended to illustrate, not limit, this disclosure. Various alternatives and equivalents are possible. This disclosure is not limited to the embodiments described herein. The embodiments are also not limited to any particular type of semiconductor device. Other additions, reductions, or modifications will be apparent from this disclosure and are intended to fall within the scope of the appended claims. Furthermore, embodiments can be combined to form other embodiments.

Claims

1. A data processing system, comprising: The controller receives neural network operation processing requests from the host device. as well as The computing device with built-in memory includes multiple processing elements, and: The controller receives the input feature map and weighted filter, and Based on the weighted filter and multiple partition maps generated from the input feature map, neural network operations are performed in the multiple processing elements. The built-in memory computing device performs the neural network operation by not moving reused elements between the processing elements, wherein the reused elements are elements that constitute the partition graph and are computed at least twice during the neural network operation.

2. The data processing system of claim 1, wherein the reused element is input to one of the plurality of processing elements only once.

3. The data processing system according to claim 1, The built-in memory computing device executes the neural network operation by performing multiple loops of neural network operations, each loop of which is performed by applying the weight filter to a corresponding partition map among the multiple partition maps. The reusable element is an element that is used in at least two of the multiple cycles.

4. The data processing system according to claim 1, wherein the built-in memory computing device further generates the plurality of partitioned maps by partitioning the input feature map based on the size of the weight filter and the step size as the moving interval of the weight filter.

5. The data processing system according to claim 1, wherein the built-in memory computing device comprises: A global buffer, which stores the input feature map and the weighted filter; The computational memory includes the plurality of processing elements and performs the neural network operations by receiving the plurality of partition maps and the weight filters; as well as Scheduler: All elements of the weighted filter are stored in the processing element, and The elements of each partition graph are assigned and provided to the processing element. The scheduler allocates and provides the elements in the following manner: The new element in the partition graph that is to be used for the first time in the neural network operation is transferred from the global buffer to the corresponding processing element among the plurality of processing elements, and The reusable element is allowed to remain in the corresponding processing element among the plurality of processing elements, where the reusable element was first provided.

6. The data processing system according to claim 1, Each of the plurality of processing elements comprises a plurality of subarrays, and The built-in memory computing device further performs the neural network operation in the following manner: Among the plurality of processing elements, a processing element corresponding to the number of elements of the weighted filter is selected. The elements of the weighted filter are assigned and stored in multiple sub-arrays included in the selected processing element, and Assign and input the elements of each partition map to the selected processing element.

7. The data processing system of claim 6, wherein each of the plurality of subarrays is configured by an array of memory cells including memristor devices.

8. The data processing system according to claim 1, The built-in memory computing device performs the neural network operation on each of the plurality of partition maps generated by moving the convolution window at fixed intervals along the row or column direction within the input feature map, and The reusable elements are elements that overlap between partition maps generated by moving the input feature maps along the row direction and / or the column direction according to the convolution window.

9. A data processing system, comprising: A global buffer that stores the input feature map and weighted filters; The computational memory includes multiple processing elements and performs multiple loops of neural network operations by receiving the weighted filter and multiple partition maps generated based on the input feature map; as well as Scheduler: Among the plurality of processing elements, a processing element corresponding to the number of elements of the weighted filter is selected. All elements of the weighted filter are stored in the selected processing element, and The elements of each partition graph are assigned and stored in the selected processing element. The scheduler allocates and stores elements of each partition graph by allowing reused elements to remain in the respective single processing element where the reused element is first provided, wherein the reused element is one of the elements of the partition graph that has been computed at least twice during the neural network operation.

10. The data processing system of claim 9, wherein the scheduler further allocates and stores elements of each partition graph in the following manner: The new element in the partition graph that is to be used for the first time in the neural network operation is transferred from the global buffer to the corresponding processing element among the plurality of processing elements, and The reusable element is allowed not to move between the plurality of processing elements.

11. The data processing system according to claim 9, Each of the plurality of processing elements comprises a plurality of subarrays, and The scheduler stores all elements of the weighted filter in the selected processing element by allocating and storing the elements of the weighted filter in multiple subarrays included in the selected processing element.

12. The data processing system of claim 11, wherein each of the plurality of subarrays is configured by an array of memory cells including memristor devices.

13. The data processing system according to claim 9, The scheduler further generates the plurality of partition maps by moving the convolutional window at fixed intervals along the row or column direction within the input feature map, and The repeated element is reused in either the row direction or the column direction.

14. A method for operating a data processing system, comprising: The controller receives neural network operation processing requests from the host device. The input feature map and weighted filter are received from the controller by a built-in memory computing device including multiple processing elements; The built-in memory computing device generates multiple partition maps based on the input feature map; and The built-in memory computing device performs neural network operations based on the plurality of partition maps and the weight filters, through at least a portion of the plurality of processing elements. The execution of the neural network operation includes controlling a reused element to not move between the processing elements, the reused element being an element that constitutes the partition graph and is operated on at least twice during the neural network operation.

15. The method of claim 14, wherein performing the neural network operation further comprises inputting the reused element into one of the plurality of processing elements only once.

16. The method according to claim 14, The neural network operation is performed through multiple loops, each loop applying the weight filter to a corresponding partition map among the multiple partition maps. The reusable element is an element that is used in at least two of the multiple cycles.

17. The method of claim 14, wherein the plurality of partition maps are generated by partitioning the input feature map based on the size of the weight filter and a step size that serves as the moving interval of the weight filter.

18. The method of claim 14, wherein the control comprises: The new element in the partition graph that is to be used for the first time in the neural network operation is transferred from the buffer to the corresponding processing element among the plurality of processing elements; and The reusable element is allowed to remain in the corresponding processing element among the plurality of processing elements, where the reusable element was first provided.

19. The method of claim 14, wherein the control comprises: Select a processing element from among the plurality of processing elements that corresponds to the number of elements in the weighted filter; The elements of the weighted filter are assigned and stored in multiple sub-arrays included in each of the selected processing elements, and Assign and input the elements of each partition map to the selected processing element.

20. The method of claim 14, wherein the plurality of partition maps are generated by moving a convolutional window at fixed intervals along a row or column direction within the input feature map, and The repeated element is reused in either the row direction or the column direction.

21. A computing system, comprising: Main unit; as well as Data processing system: In response to a neural network operation processing request from the host device, multiple partition maps are generated based on the input feature map, and Neural network operations are performed in multiple processing elements based on weighted filters and the multiple partition maps. The data processing system performs the neural network operation by not moving reused elements between the processing elements, wherein the reused elements are elements that constitute the partition graph and are computed at least twice during the neural network operation.

22. The computing system of claim 21, wherein the reusable element is input to one of the plurality of processing elements only once.

23. The computing system according to claim 21, The data processing system performs the neural network operation in the following manner: The new element to be used for the first time in the neural network operation is transferred from the buffer to the corresponding processing element among the plurality of processing elements, and The reusable element is allowed to remain in the corresponding processing element among the plurality of processing elements, where the reusable element was first provided.

24. The computing system according to claim 21, Each of the plurality of processing elements comprises a plurality of subarrays, and The data processing system further performs the neural network operation in the following manner: Among the plurality of processing elements, a processing element corresponding to the number of elements of the weighted filter is selected. The elements of the weighted filter are assigned and stored in multiple sub-arrays included in the selected processing element, and The elements of the partition graph are assigned and input to the selected processing element.

25. The computing system of claim 24, wherein each of the plurality of subarrays is configured by an array of memory cells including memristor devices.

26. The computing system of claim 21, wherein the data processing system generates the plurality of partition maps by moving a convolutional window at fixed intervals along a row or column direction within the input feature map, and The repeated element is reused in either the row direction or the column direction.

27. A computing device with built-in memory, comprising: The processing element, or PE, performs convolution operations on the filter and the partitioned graph in each loop. Each PE performs a convolution operation on the assigned filter elements and the assigned graph elements; and Control unit: The filter elements are assigned from the filter to each PE. The input feature map is divided into partition maps such that some map elements are shared by two partition maps, and In each iteration, graph elements from the selected partition graph are assigned to the individual PEs. The control unit further controls the selected PE to perform convolution operations on the recirculating graph elements in the current loop without reassigning the recirculating graph elements to the selected PE. The control unit assigns the recycle graph elements to the selected PE in the previous cycle.

Images