Integrated circuits, neural network processors and devices, and data processing methods

Abstract

An integrated circuit, a neural network processor, a neural network device, and a data processing method. The integrated circuit is included in a device for performing a neural network operation, the integrated circuit comprising: a buffer configured to store feature map data in units of cells each containing at least one feature, wherein the feature map data is used in the neural network operation; and a multiplexing circuit configured to receive the feature map data from the buffer and output extracted data by extracting feature data containing one of the features included within a plurality of cells in the received feature map data, the features each corresponding to a same coordinate value.

Description

[0001] [Cross-reference to related patent applications]

[0002] This application claims the rights of Korean Patent Application No. 10-2018-0107391, filed on September 7, 2018, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present invention relates to integrated circuits (ICs), neural network processors and neural network devices, and more specifically, to ICs for extracting data necessary for performing neural network operations, a neural network processor and a neural network device. Background Technology

[0004] A neural network is a computational architecture that simulates the biological brain. Recently, with the development of neural network technology, research is actively underway to enable various electronic systems to utilize neural network devices to analyze input data and extract useful information. These neural network devices use one or more neural network models.

[0005] For complex input data, neural network devices need to perform a huge number of calculations. Therefore, in order for neural network devices to analyze input and extract information in real time, technologies that can efficiently process neural network calculations are needed.

[0006] Since neural network devices need to perform operations on complex input data, there is a need for a method and apparatus to efficiently extract the data necessary for neural network operations from a large amount of complex input data using a small number of resources and / or low power consumption. Summary of the Invention

[0007] The present invention provides a method and apparatus for efficiently extracting data necessary for performing neural network operations in integrated circuits (ICs), neural network processors, and neural network devices using a small number of resources and / or a small power consumption.

[0008] According to an aspect of the present invention, an integrated circuit is provided in an apparatus for performing neural network operations, the integrated circuit comprising: a buffer configured to store feature map data in units of cells each comprising at least one feature, wherein the feature map data is used in the neural network operation; and a multiplexing circuit configured to receive the feature map data from the buffer and output extracted data by extracting feature data of one of a plurality of features contained in the received feature map data, each feature corresponding to the same coordinate value.

[0009] The multiplexing circuit may include: a first multiplexing circuit, including a multiplexer, the multiplexer being configured to extract feature data of one of the features contained in the plurality of cells and each corresponding to a first coordinate value, and to output first data based on the extracted feature data; and a second multiplexing circuit, including a plurality of multiplexers, the plurality of multiplexers being configured to receive the first data from the first multiplexing circuit and rotate the received first data in a vertical or horizontal direction.

[0010] According to another aspect of the present invention, a neural network processor configured to perform neural network operations is provided, the neural network processor comprising: an internal memory configured to receive external data from outside the neural network processor and to store data and / or weights based on the external data; a data extraction circuit configured to receive data from the internal memory, store the received data as feature map data in units of cells, classify the features contained in the feature map data into multiple groups, and output extracted data by extracting feature data of one feature from each of the multiple groups, wherein each of the multiple groups contains features contained in multiple cells contained in the feature map data, each feature corresponding to the same coordinate value; and an arithmetic circuit configured to perform the neural network operations using the extracted data provided by the data extraction circuit.

[0011] According to another aspect of the present invention, a data processing method is provided executed by a neural network processor configured to perform neural network operations, the data processing method comprising: storing feature map data for the neural network operations in units of cells each containing at least one feature; generating first data in matrix form by extracting feature data of one of the features contained in the plurality of cells in the feature map data for a plurality of coordinate values, the features each corresponding to the same coordinate values; generating extracted data for use in the neural network operations by rearranging the rows and / or columns of the first data; and performing the neural network operations by utilizing the extracted data, wherein the execution of the neural network operations is performed by an arithmetic circuit.

[0012] According to another aspect of the present invention, a neural network device configured to perform neural network operations is provided, the neural network device comprising: at least one intellectual property (IP); and a neural network processor configured to communicate with the at least one IP via a system bus and to output an information signal by performing the neural network operations, including convolution operations, based on input data provided by the at least one IP, wherein the neural network processor is configured to divide the input data into a plurality of cells, store the input data as an input feature map, and generate extracted data for use in the convolution operations by extracting feature data of one of the features contained in the plurality of cells, each feature corresponding to a first coordinate value.

[0013] According to another aspect of the present invention, an integrated circuit is provided in an apparatus for performing neural network operations using extracted data extracted from feature map data, the integrated circuit comprising: a buffer configured to store the feature map data in units of cells each containing at least one feature; a first multiplexing circuit configured to extract first data from the feature map data using a number of multiplexers less than or equal to the number of the at least one feature contained in each cell, wherein the first data includes all the feature data contained in the extracted data; and a second multiplexing circuit configured to generate the extracted data by rearranging the rows and / or columns of the first data using a number of multiplexers less than or equal to the sum of the number of rows and the number of columns of the first data. Attached Figure Description

[0014] An exemplary embodiment of the invention will be more clearly understood by reading the following detailed description in conjunction with the accompanying drawings, in which:

[0015] Figure 1 This is a block diagram of a neural network device according to a conceptual embodiment of the present invention.

[0016] Figure 2 This is a block diagram of a neural network device according to a conceptual embodiment of the present invention.

[0017] Figure 3 The structure of a convolutional neural network is shown as an example of a neural network structure.

[0018] Figure 4A and Figure 4B This is a schematic diagram used to explain the convolution operation of neural networks.

[0019] Figure 5 This is a block diagram of a neural network integrated circuit (IC) according to a conceptual embodiment of the present invention.

[0020] Figure 6 This is a block diagram of a data extraction circuit according to a conceptual embodiment of the present invention.

[0021] Figure 7A The feature map data and the extracted data are shown to explain the data extraction process according to a conceptual embodiment of the present invention.

[0022] Figure 7B The coordinate values ​​of the cells are shown to explain the data extraction process according to a conceptual embodiment of the present invention.

[0023] Figure 7C The cell and the first multiplexing circuit are shown to explain the data extraction process according to a conceptual embodiment of the present invention.

[0024] Figure 8A and Figure 8B The data is presented in matrix form to explain the rotation of the first data.

[0025] Figure 9A and Figure 9B This is a block diagram of a second multiplexing circuit according to a conceptual embodiment of the present invention.

[0026] Figure 10 This is a block diagram of a second multiplexing circuit according to a conceptual embodiment of the present invention.

[0027] Figure 11 This is a flowchart of a data processing method performed by a neural network device according to a conceptual embodiment of the present invention.

[0028] Figure 12 The feature map data and the extracted data are shown to explain the data extraction process according to a conceptual embodiment of the present invention.

[0029] Figures 13A to 13C This is a block diagram of a multiplexing circuit according to a conceptual embodiment of the present invention.

[0030] Figures 14A to 14C The data is presented in matrix form to explain the various data points according to the conceptual embodiments of the present invention. Figures 13A to 13C The data selection and extraction method performed by the multiplexing circuit shown is illustrated.

[0031] [Explanation of Symbols]

[0032] 10: Neural network device;

[0033] 100: Neural network integrated circuits, neural network processing units;

[0034] 120: Internal memory;

[0035] 122: Data storage device;

[0036] 124: Weighted memory;

[0037] 140: Data extraction circuit;

[0038] 141: Data buffer;

[0039] 142, 142a, 142b, 142c, 143, 143a, 143b, 143c, 144, 144a, 144b, 144c: Multiplexing circuits;

[0040] 145, 145a, 145b: Vertical rotation circuit;

[0041] 146, 146a, 146b: Horizontal rotation circuit;

[0042] 147a, 147b, 147c, 148b: Data selection circuit;

[0043] 149: Extraction controller;

[0044] 160: Arithmetic circuits;

[0045] 200: Random Access Memory;

[0046] 201: Input feature map;

[0047] 202: Kernel;

[0048] 203, 230: Output feature maps;

[0049] 210: Input feature map;

[0050] 211, 212, 213, Data_ext: Extracted data;

[0051] 220: Raw kernel;

[0052] 231, 232, 233: Feature data;

[0053] 300: Processor;

[0054] 400: Memory;

[0055] 500: Sensor module;

[0056] CTRL_Mul: Multiplexer control signal;

[0057] Cell_1~Cell_16, Cell_i: cell;

[0058] CTRL_11, CTRL_12, CTRL_44, CTRL_R1~CTRL_R4, CTRL_C1~CTRL_C4: control signals;

[0059] CL: Category;

[0060] D1_11, D1_12, D1_44: Extracted data;

[0061] Data, D1, D1_VR, D1_HR, D1_SEL, D11~D44: Data;

[0062] D_FM: Feature map data;

[0063] D: Depth;

[0064] FM1~FMn: Feature maps;

[0065] H: Height;

[0066] IP1~IP3: Intellectual Property Rights;

[0067] IS: Information signal;

[0068] L1~Ln: Layers;

[0069] MUX11, MUX12, MUX44, MUX_R1~MUX_R4, MUX_C1~MUX_C4: Multiplexers;

[0070] NN: Neural Network;

[0071] PW: Pooling window;

[0072] REC: Identification signal;

[0073] S100~S400: Operation

[0074] W: Width;

[0075] WM: Weighted graph. Detailed Implementation

[0076] The concept of the invention will be more fully explained below with reference to the accompanying drawings, in which exemplary embodiments of the concept of the invention are shown.

[0077] Figure 1This is a block diagram of a neural network device 10 according to a conceptual embodiment of the present invention. The neural network device 10 may include one or more intellectual property (IP) and / or neural network integrated circuits (ICs) 100. For example, the neural network device 10 may include first IP IP1 to third IP IP3 and neural network IC 100. The neural network device 10 may be designed to perform various functions in a semiconductor system, and for example, the neural network device 10 may be an application processor. The neural network device 10 may analyze input data in real time and extract valid information based on a neural network, and based on the extracted valid information, may determine a situation or control at least one of the IPs. For example, the neural network device 10 may be used in drones, robotic devices (e.g., advanced driver assistance systems (ADAS)), smart televisions (TVs), smartphones, medical devices, mobile devices, image display devices, measuring devices, Internet of Things (IoT) devices, etc., and additionally, may be used in at least one of various electronic devices.

[0078] The neural network device 10 may include various IPs. For example, the IPs may include a processing unit, multiple cores included in the processing unit, a multi-format codec (MFC), a video module (e.g., a camera interface, a Joint Photographic Experts Group (JPEG) processor, a video processor, or a mixer), a three-dimensional (3D) graphics core, an audio system, a driver, a display driver, volatile memory, non-volatile memory, a memory controller, an input and output interface block, and / or a cache memory. Each of the first IPs IP1 through IP3 may include at least one of various IPs.

[0079] Examples of technologies used to connect various IPs involve system bus-based connectivity methods. For instance, the Advanced Microcontroller Bus Architecture (AMBA) protocol for Advanced RISC (Reduced Instruction Set Computer) Machines (ARM) can be used as the standard bus specification. Examples of bus types for the AMBA protocol include Advanced High-Performance Bus (AHB), Advanced Peripheral Bus (APB), Advanced eXtensible Interface (AXI), AXI4, and AXI Coherency Extensions (ACE). AXI, among these bus types, is the interface protocol between IPs and can provide features such as multiple outstanding address function and data interleaving function. In addition to the bus types mentioned above, other types of protocols (such as SONICs Inc.'s uNetwork, IBM's CoreConnect, and OCP-IP's Open Core Protocol) are also applicable to the system bus.

[0080] The neural network IC 100 can generate a neural network, train (or learn) the neural network, perform computations based on received input data and generate information signals based on the results of the computations, or retrain the neural network. The neural network can include various types of models, such as convolutional neural networks (CNNs) (e.g., GoogleNet, AlexNet, and VGG networks), a region with a convolutional neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), deconvolutional networks, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory (LSTM) networks, and classification networks, but embodiments are not limited thereto. The neural network IC 100 may include one or more processors for performing computations according to a model of the neural network. The neural network IC 100 may also include a dedicated memory (not shown) for storing programs corresponding to the model of the neural network. The neural network IC 100 may be referred to as a neural network processing device, neural network processor, neural network processing unit (NPU), etc.

[0081] The neural network IC 100 can receive various input data from the one or more IPs via a system bus and can generate information signals based on the input data. For example, the neural network IC 100 can generate information signals by performing neural network operations on the input data, and the neural network operations may include convolution operations. See below for further details. Figure 4A and Figure 4BThe convolution operation of the neural network IC 100 will be described in detail below. The information signal generated by the neural network IC 100 may include, for example, one of the following various recognition signals: voice recognition signal, object recognition signal, image recognition signal, and / or biometric recognition signal. For example, the neural network IC 100 may receive frame data contained in a video stream as input data, and may generate a recognition signal about an object contained in an image represented by the frame data based on the frame data. However, the embodiments are not limited to this, and the neural network IC 100 may receive various input data and may generate recognition signals based on the input data.

[0082] In a neural network device 10 according to a conceptual embodiment of the present invention, a neural network IC 100 can store input feature map data in a buffer in units of cells based on input data provided by the one or more IPs. Each cell may contain at least one feature. The neural network IC 100 can generate input extracted data by extracting feature data of one of the features corresponding to the same coordinate value and respectively contained in a plurality of cells included in the input feature map data. In other words, the neural network IC 100 can generate a value corresponding to the first coordinate value by extracting feature data of one of the features respectively contained in the plurality of cells and corresponding to a first coordinate value of the input extracted data. According to an embodiment, the neural network IC 100 can generate first data based on the extracted feature data, and can generate input extracted data by rotating the first data in the vertical and / or horizontal directions. The neural network IC 100 can perform convolution by multiplying the input extracted data by weight values. As described above, the neural network IC 100 can extract the data necessary for computation by using a small multiplexer to extract feature data from one of the features respectively contained in the plurality of cells, each feature corresponding to the same coordinate value. Therefore, the data extraction speed of the neural network IC 100 can be improved, and the power consumption of data extraction can be reduced. Furthermore, the operating speed of the neural network device 10 can be improved, or the neural network device 10 can consume less power. Specifically, since the neural network IC 100 can be implemented using a smaller number of multiplexers compared to conventional techniques, the area occupied by the multiplexers can be reduced when the neural network IC 100 is implemented as a chip, which is therefore advantageous in terms of design. Figure 2 This is a block diagram of a neural network device 10 according to an embodiment of the present invention. Specifically, Figure 2 yes Figure 1 An exemplary embodiment of the neural network device 10 is shown. Therefore, the neural network device 10 and the reference datum are not discussed further herein. Figure 1 Explanations that are identical to those provided will be repeated.

[0083] The neural network device 10 may include random access memory (RAM) 200, processor 300, memory 400, and / or sensor module 500.

[0084] According to an embodiment, the neural network IC 100 may be an NPU.

[0085] RAM 200 can temporarily store programs, data, or instructions. For example, under the control of processor 300 or depending on the boot code, programs and / or data stored in memory 400 can be temporarily stored in RAM 200. RAM 200 can be implemented using dynamic random access memory (DRAM) or static random access memory (SRAM).

[0086] Processor 300 controls the overall operation of neural network device 10. For example, processor 300 may be a central processing unit (CPU). Processor 300 may include a single processor core or multiple processor cores. Processor 300 can process or execute programs and / or data stored in RAM 200 and memory 400. For example, processor 300 can control the functions of neural network device 10 by executing programs stored in memory 400.

[0087] Memory 400 is a storage device for storing data, and can store, for example, an operating system (OS), various programs, and various types of data. Memory 400 may be, but is not limited to, DRAM. Memory 400 may include at least one of volatile memory and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memory may include DRAM, SRAM, synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. According to an embodiment, the memory 400 may include at least one of the following: hard disk drive (HDD), solid state drive (SSD), compact flash (CF), secure digital (SD) card, micro-secure digital (Micro-SD) card, mini-secure digital (Mini-SD) card, extreme digital (xD) card, and memory stick.

[0088] Sensor module 500 can collect information about the vicinity of neural network device 10. Sensor module 500 can sense or receive image signals from outside neural network device 10 and convert the sensed or received image signals into image data, i.e., image frames. For this purpose, sensor module 500 may include sensing devices, i.e., at least one of various sensing devices such as an imaging device, an image sensor, a light detection and ranging (LIDAR) sensor, an ultrasonic sensor, and an infrared sensor, or may receive sensing signals from sensing devices. According to an embodiment, sensor module 500 can provide image frames to neural network IC 100. For example, sensor module 500 may include an image sensor and can capture images of the external environment of neural network device 10 to generate a video stream, and can sequentially provide consecutive image frames of the video stream to neural network IC 100.

[0089] In a neural network device 10 according to a conceptual embodiment of the present invention, a neural network IC 100 can store input feature map data in a buffer on a cell-by-cell basis based on input data provided by one or more IPs. Each cell may contain at least one feature. The neural network IC 100 can generate input extraction data by extracting feature data of one of the features corresponding to the same coordinate value and contained in a plurality of cells included in the input feature map data. In other words, the neural network IC 100 can generate a value corresponding to the first coordinate value by extracting feature data of one of the features respectively contained in the plurality of cells and corresponding to a first coordinate value of the input extraction data. According to an embodiment, the neural network IC 100 can generate first data based on the extracted feature data, and can generate input extraction data by rotating the first data in a vertical and / or horizontal direction. The neural network IC 100 can perform convolution by multiplying the input extraction data by a weight value. As described above, the neural network IC 100 can utilize a small multiplexer to extract the data necessary for computation by extracting feature data of one of the features corresponding to the same coordinate value and contained in a plurality of cells. Therefore, the data extraction speed of the neural network IC 100 can be improved, and the power consumption of data extraction can be reduced. Furthermore, the operating speed of the neural network device 10 can be improved, or the neural network device 10 can consume less power. Specifically, since the neural network IC 100 can be implemented using a smaller number of multiplexers compared to conventional techniques, the area occupied by the multiplexers can be reduced when the neural network IC 100 is implemented as a chip, which is therefore advantageous in terms of design.

[0090] Figure 3The structure of a convolutional neural network is shown as an example of a neural network structure. The neural network NN may include multiple layers L1 to Ln. Each of the multiple layers L1 to Ln may be a linear layer or a non-linear layer. According to an embodiment, at least one linear layer is coupled to at least one non-linear layer, and the result of the coupling can be represented by a single layer. For example, the linear layer may include a convolutional layer and a fully connected layer, and the non-linear layer may include a pooling layer and an activation layer.

[0091] For example, the first layer L1 can be a convolutional layer, the second layer L2 can be a pooling layer, and the nth layer can be a fully connected layer that serves as the output layer. The neural network NN may further include activation layers and may further include layers configured to perform other kinds of operations.

[0092] Each of the plurality of layers L1 to Ln may receive a feature map from input data (e.g., an image frame) or a feature map generated from the previous layer as an input feature map, and perform operations on the input feature map to produce an output feature map or a recognition signal REC. In an embodiment, the feature map represents data in which various features of the input data are expressed. The first feature map FM1, the second feature map FM2, the third feature map FM3 to the nth feature map FMn may, for example, each have a two-dimensional (2D) matrix form or a three-dimensional (3D) matrix (or tensor) form. Each of the first feature map FM1, the second feature map FM2, the third feature map FM3 to the nth feature map FMn may have a width W (or column), a height H (or row), and a depth D. These parameters may correspond to the x-axis, y-axis, and z-axis in a coordinate system, respectively. In an embodiment, the depth D may be represented by the number of channels.

[0093] The first layer L1 performs convolution on the first feature map FM1 and the weight map WM to produce a second feature map FM2. The weight map WM can filter the first feature map FM1 and can be represented by a filter or kernel. The depth of the weight map WM (i.e., the number of channels in the weight map WM) is the same as the depth of the first feature map FM1 (i.e., the number of channels), and convolution can be performed on the same channels of the weight map WM and the first feature map FM1. The first input feature map FM1 can be used as a sliding window to shift the weight map WM in a crosswise manner. The shift amount can be represented by the "stride length" or "stride". During each shift, the weights contained in the weight map WM are multiplied by all the feature data of the portion of the first feature map FM1 that overlaps with the weight map WM, and then summed. The feature data of the portion of the first feature map FM1 that overlaps with the weight values ​​contained in the weight map WM can be referred to as the extracted data. When convolving the first feature map FM1 with the weight map WM, one channel of the second feature map FM2 can be generated. Although Figure 3 A weight map WM is shown, but in practice, multiple weight maps can be convolved with the first feature map FM1, and thus, multiple channels of the second feature map FM2 can be generated. In other words, the number of channels in the second feature map FM2 can correspond to the number of weight maps WM.

[0094] The second layer L2 can generate a third feature map FM3 by using pooling to modify the spatial size of the second feature map FM2. Pooling can be represented by sampling or downsampling. A 2D pooling window PW can be shifted on the second feature map FM2 in units of the size of the pooling window PW, and the maximum value (or, alternatively, the average value) of the feature data in the portion of the second feature map FM2 that overlaps with the pooling window PW can be selected. Therefore, a third feature map FM3 with a modified spatial size can be generated from the second feature map FM2. The number of channels in the third feature map FM3 is the same as the number of channels in the second feature map FM2.

[0095] The nth layer Ln combines the features of the nth feature map FMn to classify the input data into category CL. The nth layer can generate a recognition signal REC corresponding to the category. According to an embodiment, the input data may correspond to frame data contained in a video stream, and the nth layer Ln can identify the object and generate a recognition signal REC corresponding to the identified object by extracting the category corresponding to the object contained in the image represented by the frame data based on the nth feature map FMn provided from the previous frame.

[0096] Figure 4A and Figure 4BThis is a schematic diagram used to explain the convolution operation of neural networks.

[0097] Reference Figure 4A The input feature map 201 may contain D channels, and each channel of the input feature map may have a column size of H and a row size of W (where D, H, and W are natural numbers). Each kernel 202 may have a column size of R and a row size of S, and kernel 202 may contain a number of channels corresponding to the number D (or depth) of the input feature map 201, where R and S are natural numbers. The output feature map 203 may be generated by a 3D convolution operation between the input feature map 201 and kernel 202, and may contain Y channels according to the 3D convolution operation.

[0098] For reference Figure 4B This describes the process of generating an output feature map through a 2D convolution operation between an input feature map and a kernel, and this operation can be performed on the input feature map 201 and the kernel 202 of all channels. Figure 4B The 2D convolution operation shown can thus produce the output feature map 203 for all channels.

[0099] Reference Figure 4B For ease of explanation, it is assumed that the input feature map 210 has a size of 6×6, the original kernel 220 has a size of 3×3, and the output feature map 230 has a size of 4×4, but the embodiments are not limited to this. Neural networks can be implemented using feature maps of various sizes and kernels of various sizes. The values ​​defined in the input feature map 210, the original kernel 220, and the output feature map 230 are all exemplary values, and the embodiments of the present invention are not limited to these.

[0100] The original kernel 220 can perform convolution operations while sliding across the input feature map 210 in 3×3 window units. The convolution operation can be represented by calculating each feature data of the output feature map 230 by multiplying each feature data of the window of the input feature map 210 by the weight value at the corresponding position on the original kernel 220, and then summing the results of the multiplications. The feature data multiplied by the weight values ​​contained in the window of the input feature map 210 can be referred to as the extracted data from the input feature map 210. Specifically, the original kernel 220 can first undergo convolution together with the first extracted data 211 of the input feature map 210. In other words, the feature data 1, 2, 3, 4, 5, 6, 7, 8, and 9 of the first extracted data 211 can be multiplied by the weight values ​​of the original kernel 220: -1, -3, 4, 7, -2, -1, -5, 3, and 1, respectively. Therefore, -1, -6, 12, 28, -10, -6, -35, 24, and 9 can be obtained. Next, the obtained values ​​-1, -6, 12, 28, -10, -6, -35, 24, and 9 can be added together to obtain 15. The feature data 231 in the first row and first column of the output feature map 230 can then be determined as 15. The feature data 231 in the first row and first column of the output feature map 230 corresponds to the first extracted data 211. Similarly, the second extracted data 212 of the input feature map 210 can be convolved with the original kernel 220, and therefore, the feature data 232 in the first row and second column of the output feature map 230 can be determined as 4. Finally, the sixteenth extracted data 213 (which is the last extracted data of the input feature map 210) can be convolved with the original kernel 220, and thus, the feature data 233 in the fourth row and fourth column of the output feature map 230 can be determined as 11.

[0101] In other words, convolution of a single input feature map 210 with a single original kernel 220 can be achieved by repeatedly multiplying the extracted data of the input feature map 210 with the weight values ​​of the original kernel 220 and adding the results of the multiplication. As a result of the convolution, an output feature map 230 can be generated.

[0102] Reference Figure 1 , Figure 4A and Figure 4BThe neural network IC 100 can generate an input feature map based on input data received from the one or more IPs, and can extract data directly multiplied by weight values ​​from the input feature map. For this purpose, the neural network IC 100 may include at least one multiplexer (MUX), and can utilize the at least one multiplexer to extract data directly multiplied by weight values ​​from each input feature map data contained in the input feature map. The data directly multiplied by weight values ​​may be referred to as the extracted data. For ease of explanation, each of the values ​​of a plurality of features contained in the feature map will now be referred to as feature data, and the plurality of feature data contained in the feature map will now be referred to as feature map data. Feature data of a feature is considered to refer to the feature value of the feature. An apparatus and method for extracting extracted data directly used in computation from a feature map or feature map data according to an embodiment of the present invention will now be described with reference to the following figures.

[0103] Figure 5 This is a block diagram of a neural network IC 100 according to a conceptual embodiment of the present invention. The neural network IC 100 may include an internal memory 120, a data extraction circuit 140, and / or an arithmetic circuit 160. Further details are not provided herein. Figure 5 The neural network IC100 shown is compared with the reference. Figure 1 and Figure 2 Explanations that are identical to those provided will be repeated.

[0104] Internal memory 120 can receive external data from outside the neural network IC 100. This external data may also be referred to as input data. Internal memory 120 can store the external data, various data necessary for computation, and weight values. For this purpose, internal memory 120 may include a data memory 122 for storing the various data, and / or a weight memory 124 for storing the weight values. Data memory 122 and weight memory 124 may be configured as separate hardware, but embodiments are not limited to this. For example, data memory 122 and weight memory 124 may represent memories corresponding to different areas within a single piece of hardware. Various types of memory (e.g., DRAM, SRAM, and SDRAM) can be used to implement each of data memory 122 and weight memory 124.

[0105] Data extraction circuit 140 can generate extracted data Data_ext based on data Data stored in data memory 122. Data Data can indicate feature map data, and the extracted data Data_ext can indicate the data necessary for calculation in each data item contained in the feature map data. Data extraction circuit 140 can store data Data as feature map data in units of cells. For example, data extraction circuit 140 can include a buffer that stores data Data as feature map data in units of cells. Each cell can contain at least one feature. For example, a cell can have a size of four features × four features. Data extraction circuit 140 can generate extracted data Data_ext by extracting feature data of one of the features corresponding to the same coordinate value and respectively contained in multiple cells in the feature map data. In other words, data extraction circuit 140 can extract feature data of one of the features respectively contained in the multiple cells and corresponding to a first coordinate value, and can generate first data by extracting each feature data corresponding to all coordinate values. According to an embodiment, the data extraction circuit 140 generates extracted data Data_ext by rotating first data in a vertical and / or horizontal direction. For example, the data extraction circuit 140 can rotate the first data vertically by changing at least a portion of the order of the rows of the first data, and can rotate the first data horizontally by changing at least a portion of the order of the columns of the first data. The amount of rotation of the first data in the vertical and / or horizontal direction can be determined based on the position of the extracted data Data_ext on the feature map data. In other words, the data extraction circuit 140 may not change the order of the rows and columns of the first data depending on the position of the extracted data Data_ext on the feature map data. The data extraction circuit 140 can provide the extracted data Data_ext to the arithmetic circuit 160.

[0106] Arithmetic circuit 160 can receive extracted data Data_ext from data extraction circuit 140 and can perform calculations based on the extracted data Data_ext. The calculations may include at least one of various calculations such as multiplication, addition, and XOR operation. According to an embodiment, arithmetic circuit 160 can perform a convolution operation of neural network IC 100 by multiplying the weight values ​​stored in weight memory 124 by the extracted data Data_ext corresponding to the weight values, and then adding the results of the multiplication. Arithmetic circuit 160 can output an information signal IS outside neural network IC 100 through at least one calculation.

[0107] Figure 6This is a block diagram of a data extraction circuit 140 according to an embodiment of the present invention. The data extraction circuit 140 may include a data buffer 141, a multiplexing circuit 142, and / or an extraction controller 149. Further details are not provided herein. Figure 6 The data extraction circuit 140 shown is related to the reference. Figure 5 Explanations that are identical to those provided will be repeated.

[0108] Data buffer 141 can store feature map data D_FM based on data Data received from outside data extraction circuit 140. According to an embodiment, data buffer 141 can store feature map data D_FM in units of cells, each containing at least one feature. In other words, data buffer 141 can classify feature map data D_FM according to multiple cells and store the feature map data D_FM as said multiple cells. According to an embodiment, data extraction circuit 140 may further include a processor (not shown) and a memory (not shown), and the processor executes instructions stored in the memory to cause data extraction circuit 140 to perform a certain operation. For example, data extraction circuit 140 can store address information of each of the multiple cells contained in feature map data D_FM in the memory, and obtain each feature data contained in the multiple cells by accessing the multiple cells based on the address information stored in the memory.

[0109] The multiplexing circuit 142 can output the extracted data Data_ext by retrieving the necessary data for computation from the feature map data D_FM stored in the data buffer 141. For this purpose, the multiplexing circuit 142 may include at least one MUX. According to an embodiment, the multiplexing circuit 142 may include a first multiplexing circuit 143 and / or a second multiplexing circuit 144.

[0110] The first multiplexing circuit 143 can extract feature data from one of the features contained in the plurality of cells (each feature corresponding to the same coordinate value) included in the feature map data D_FM, and can generate first data D1 by extracting each feature data corresponding to all coordinate values. The operation of generating first data D1 performed by the first multiplexing circuit 143 can be referred to as data extraction, and will be discussed later. Figures 7A to 7CTo illustrate data extraction, according to an embodiment, the first multiplexing circuit 143 may include a plurality of multiplexers, each of which selects an input signal from a number of input signals corresponding to the number of cells contained in the feature map data D_FM. For example, when the feature map data D_FM contains N (where N is a natural number) cells, the first multiplexing circuit 143 may include a plurality of N:1 multiplexers. According to an embodiment, the first multiplexing circuit 143 may include a number of "M" multiplexers corresponding to the number of features contained in each cell. For example, when each cell contains M (where M is a natural number) features, the first multiplexing circuit 143 may include M multiplexers. In other words, when the feature map data D_FM contains N cells and each cell contains M features,

[0111] The first multiplexing circuit 143 may include M N:1 multiplexers.

[0112] According to an embodiment, the first multiplexing circuit 143 can classify the features contained in the feature map data D_FM into multiple groups. In an embodiment, each of the multiple groups may contain features included in the multiple cells contained in the feature map data D_FM, and each feature corresponds to the same coordinate value. The first multiplexing circuit 143 can extract feature data of one feature from each of the multiple groups, and can generate first data D1 based on the extracted feature data. According to an embodiment, the first data D1 may be data in matrix form.

[0113] The second multiplexing circuit 144 can generate the extracted data Data_ext based on the first data D1 provided by the first multiplexing circuit 143. For example, the second multiplexing circuit 144 can generate the extracted data Data_ext by rotating the first data D1 in a vertical and / or horizontal direction. For example, the second multiplexing circuit 144 can rotate the first data D1 vertically by changing the order of its rows, and can rotate the first data D1 horizontally by changing the order of its columns. According to an embodiment, the second multiplexing circuit 144 can generate the extracted data Data_ext by rotating the first data D1 vertically and then rotating the vertically rotated first data horizontally. However, the embodiments are not limited to this. According to an embodiment, the second multiplexing circuit 144 can generate the extracted data Data_ext by rotating the first data D1 horizontally and then rotating the horizontally rotated first data vertically. According to an embodiment, the second multiplexing circuit 144 can be implemented using a single piece of hardware, which rotates the first data in both the vertical and horizontal directions. The operation of generating the extracted data Data_ext based on the first data D1 can be referred to as a data rotation operation, and will be discussed later. Figures 8A to 10 To elaborate on the data rotation operation in more detail, according to an embodiment, the second multiplexing circuit 144 may include a number of multiplexers corresponding to a value obtained by adding the number of rows and columns of the first data D1. For example, when the first data D1 represents a K×L matrix (where K and L are natural numbers), the second multiplexing circuit 144 may include (K+L) multiplexers. Specifically, the second multiplexing circuit 144 may include K multiplexers for vertical rotation and L multiplexers for horizontal rotation.

[0114] The extraction controller 149 can generate at least one multiplexer control signal CTRL_Mul for controlling at least one multiplexer included in the multiplexing circuit 142, and can provide the at least one multiplexer control signal CTRL_Mul to the multiplexing circuit 142. The at least one multiplexer included in the multiplexing circuit 142 can select one input signal from a plurality of input signals based on the at least one multiplexer control signal CTRL_Mul. The extraction controller 149 can be implemented using dedicated hardware (e.g., analog circuitry), or the operation of the extraction controller 149 can be performed by a processor included in the data extraction circuit 140.

[0115] Figure 7A The feature map data D_FM and the extracted data Data_ext are shown to explain the data extraction process according to a conceptual embodiment of the present invention. For ease of explanation, Figure 7A An example is shown in which the feature map data D_FM contains 16 cells, each containing 16 features; however, the number of cells and the number of features are merely examples. Therefore, embodiments of the present invention are not limited thereto. Reference will now be made to... Figure 5 and Figure 6 To explain Figure 7A .

[0116] When the neural network IC 100 performs neural network operations (e.g., convolution) based on the data feature map, the arithmetic circuit 160 may require extracted data Data_ext from the feature map data D_FM. In an embodiment, the data extraction circuit 140 may extract the extracted data Data_ext from the feature map data D_FM. According to an embodiment, the size of the extracted data Data_ext may be less than or equal to the size of each cell.

[0117] The feature map data D_FM may contain cells 1 through 16 (Cell_16). The feature map data D_FM may be stored in data buffer 141 as cells 1 through 16. A portion of the extracted data Data_ext, necessary for neural network operations, is contained in cell 1, another portion in cell 2, another portion in cell 5, and another portion in cell 6. Reference will now be made to… Figure 7B and Figure 7C To explain from Figure 7A Feature map data D_FM extraction shown Figure 7A The process of extracting the Data_ext shown.

[0118] Figure 7B The coordinates of cell_i are shown to explain the data extraction process according to a conceptual embodiment of the present invention. Cell_i is... Figure 7A The diagram shows representative instances of cells 1 through 16. For ease of explanation, Figure 7B An embodiment is shown in which cell_i contains 16 features. However, the embodiments are not limited to this.

[0119] A cell, Cell_i, can be a matrix containing multiple rows and columns. Cell_i can contain multiple features, each corresponding to a coordinate value determined based on a row and a column. In this embodiment, for ease of explanation, the coordinate value of a feature located in the i-th row and j-th column of Cell_i is expressed as (i, j) (where i and j are natural numbers less than or equal to 4). Since Cell_i contains 16 features, it can contain 16 features with 16 distinct coordinate values.

[0120] Figure 7C The cell and the first multiplexing circuit 143 are shown to explain the data extraction process according to an embodiment of the present invention. For example, Figure 7C Showing according to, for example Figure 7A The embodiment extracts the extracted data Data_ext from the feature map data D_FM using a first multiplexing circuit 143. Reference will now be made to... Figure 7A To explain Figure 7C .

[0121] The features contained in the feature map data D_FM can be classified into multiple groups. For example, the (1,1) group can contain features corresponding to the (1,1) coordinate values ​​and contained in cells 1 through 16. For example, the (1,2) group can contain features corresponding to the (1,2) coordinate values ​​and contained in cells 1 through 16. For example, the (4,4) group can contain features corresponding to the (4,4) coordinate values ​​and contained in cells 1 through 16.

[0122] The first multiplexing circuit 143 can generate first data by extracting one data point from each of the plurality of groups. For this purpose, the first multiplexing circuit 143 may include a plurality of multiplexers corresponding to the plurality of groups. According to an embodiment, each of the plurality of multiplexers may be a multiplexer that selects one input signal from a number of input signals corresponding to the number of cells contained in the feature map data D_FM. According to an embodiment, the first multiplexing circuit 143 may include a number of multiplexers corresponding to the number of features contained in each cell. For example, in Figure 7A In the illustrated embodiment, the first multiplexing circuit 143 may include sixteen 16:1 multiplexers. According to an embodiment, each of the plurality of multiplexers may extract data from one of the plurality of cells based on the position of the extracted data Data_ext on the feature map data D_FM.

[0123] The multiplexer MUX11 can extract feature data from one of the features corresponding to the (1,1) coordinate value based on the control signal CTRL_11 to output the first data D1_11 corresponding to the (1,1) coordinate value. Specifically, in Figure 7A In the illustrated embodiment, multiplexer MUX11 can output first data D1_11 corresponding to the (1,1) coordinate value by extracting feature data of the feature contained in the sixth cell Cell_6 from the feature corresponding to the (1,1) coordinate value. The control signal CTRL_11 can be determined based on the position of the extracted data Data_ext on the feature map data D_FM. Similarly, multiplexer MUX12 can output first data D1_12 corresponding to the (1,2) coordinate value by extracting feature data of one of the features corresponding to the (1,2) coordinate value based on the control signal CTRL_12. Specifically, in Figure 7A In the illustrated embodiment, the multiplexer MUX12 can output the first data D1_12 corresponding to the (1,2) coordinate values ​​by extracting feature data of the features contained in the fifth cell Cell_5 from the features corresponding to the (1,2) coordinate values. The control signal CTRL_12 can be determined based on the position of the extracted data Data_ext on the feature map data D_FM. Figure 6 The extraction controller 149 shown provides control signals CTRL_11, CTRL_12 to CTRL_44. For example, control signals CTRL_11, CTRL_12 to CTRL_44 may be included in... Figure 6 The multiplexer control signal CTRL_Mul is shown.

[0124] The first multiplexing circuit 143 can output first data based on each extracted data D1_11, D1_12 to D1_44.

[0125] Figure 8A and Figure 8B The data is presented in matrix form to explain the rotation of the first data D1. For ease of explanation, Figure 8A and Figure 8B Showing with Figure 7A The data corresponding to the illustrated embodiment. Now, refer to... Figure 6 To explain Figure 8A and Figure 8B .

[0126] Reference Figure 8AThe first data D1 generated by the first multiplexing circuit 143 contains all the feature data contained in the extracted data Data_ext, which is necessary for calculation. However, the order of the rows and columns of the first data D1 may differ from the order of the rows and columns of the extracted data Data_ext. Therefore, the second multiplexing circuit 144 can generate the extracted data Data_ext by rearranging the rows and / or columns of the matrix-formed first data D1. In other words, the second multiplexing circuit 144 can generate the extracted data Data_ext by rotating the first data D1 in the vertical and / or horizontal directions.

[0127] Reference Figure 8B The first data D1 can be transformed into the extracted data Data_ext through vertical and horizontal rotation. For ease of explanation, Figure 8B This illustrates a case where a horizontal rotation is performed after a vertical rotation. However, the order of vertical and horizontal rotations is not limited to this. For example, with... Figure 8B In contrast, a vertical rotation can be performed after a horizontal rotation. The second multiplexing circuit 144 can generate vertically rotated first data D1_VR by rotating the first data D1 in a vertical direction. In other words, the second multiplexing circuit 144 can generate vertically rotated first data D1_VR by changing the order of the rows of the first data D1. Subsequently, the second multiplexing circuit 144 can generate extracted data Data_ext by rotating the vertically rotated first data D1_VR horizontally. In other words, the second multiplexing circuit 144 can generate extracted data Data_ext by changing the order of the columns of the vertically rotated first data D1_VR. To perform vertical and / or horizontal rotation, the second multiplexing circuit 144 may include at least one multiplexer. Reference will now be made to... Figure 9A , Figure 9B and Figure 10 An embodiment of the configuration of the second multiplexing circuit 144 is described below.

[0128] Figure 9A and Figure 9B This is a block diagram of the second multiplexing circuits 144a and 144b according to a conceptual embodiment of the present invention.

[0129] Reference Figure 9AThe second multiplexing circuit 144a may include a vertical rotation circuit 145a and / or a horizontal rotation circuit 146a. The vertical rotation circuit 145a generates vertically rotated first data D1_VR by rotating the first data D1 vertically, and the horizontal rotation circuit 146a generates extracted data Data_ext by rotating the vertically rotated first data D1_VR horizontally. Therefore, each of the vertical rotation circuit 145a and the horizontal rotation circuit 146a may include at least one multiplexer.

[0130] Reference Figure 9B The second multiplexing circuit 144b may include a horizontal rotation circuit 146b and / or a vertical rotation circuit 145b. The horizontal rotation circuit 146b generates horizontally rotated first data D1_HR by rotating the first data D1 horizontally, and the vertical rotation circuit 145b generates extracted data Data_ext by rotating the horizontally rotated first data D1_HR vertically. Therefore, each of the horizontal rotation circuit 146b and the vertical rotation circuit 145b may include at least one multiplexer.

[0131] According to the embodiments, it can be used Figure 9A The second multiplexing circuit 144a shown is or Figure 9B The second multiplexing circuit 144b shown is used to implement this. Figure 6 The second multiplexing circuit 144 is shown. However, as referenced above... Figure 6 The above, Figure 6 The second multiplexing circuit 144 shown is not limited to Figure 9A or Figure 9B The illustrated embodiment. For example, it can be implemented using a single piece of hardware. Figure 6 The second multiplexing circuit 144 shown above generates the extracted data Data_ext by rotating the first data D1 in both the vertical and horizontal directions.

[0132] Figure 10 This is a block diagram of a second multiplexing circuit 144 according to an embodiment of the present invention. Specifically, Figure 10 Showing has Figure 9A The second multiplexing circuit 144 is configured as shown. The same rules apply. Figure 9B The embodiment shown. Referring now to... Figure 7A and Figure 9A To explain Figure 10 .

[0133] The vertical rotation circuit 145 can generate vertically rotated first data D1_VR by changing the order of the rows of the first data D1. For this purpose, the vertical rotation circuit 145 may include a number of multiplexers corresponding to the number of rows of the first data D1. The multiplexer MUX_R1 can output the first row of the vertically rotated first data D1_VR by selecting one of the rows of the first data D1 based on the control signal CTRL_R1. (See reference...) Figure 10 Multiplexer MUX_R1 can output the first row of vertically rotated first data D1_VR by selecting the data D1[2,1:4] corresponding to the second row of the first data D1. Similarly, multiplexer MUX_R4 can output the fourth row of vertically rotated first data D1_VR by selecting one of the rows of the first data D1 based on the control signal CTRL_R4. (See reference...) Figure 10 The multiplexer MUX_R4 can output the fourth row of the vertically rotated first data D1_VR by selecting each data D1[1,1:4] corresponding to the first row of the first data D1. The vertical rotation circuit 145 can provide the vertically rotated first data D1_VR to the horizontal rotation circuit 146 based on the data output by the multiplexers MUX_R1 to MUX_R4.

[0134] The horizontal rotation circuit 146 generates the extracted data Data_ext by changing the order of the columns of the vertically rotated first data D1_VR. For this purpose, the horizontal rotation circuit 146 may include a number of multiplexers corresponding to the number of columns of the first data D1. The multiplexer MUX_C1 can output the first column of the extracted data Data_ext by selecting one of the vertically rotated columns of the first data D1_VR based on the control signal CTRL_C1. (See reference...) Figure 10 Multiplexer MUX_C1 can output the first column of the extracted data Data_ext by selecting the data D1_VR[1:4,2] corresponding to the second column of the vertically rotated first data D1_VR. Similarly, multiplexer MUX_C4 can output the fourth column of the extracted data Data_ext by selecting one of the vertically rotated first data D1_VR columns based on the control signal CTRL_C4. (See reference...) Figure 10 The multiplexer MUX_C4 can output the fourth column of the extracted data Data_ext by selecting each data D1_VR[1:4,1] corresponding to the first column of the vertically rotated first data D1_VR. The horizontal rotation circuit 146 can output the extracted data Data_ext based on the data selected by the multiplexers MUX_C1 to MUX_C4.

[0135] Figure 11 This is a flowchart of a data processing method performed by a neural network device according to a conceptual embodiment of the present invention. Reference will now be made to... Figure 5 and Figure 6 To explain Figure 11 .

[0136] In operation S100, the neural network IC 100 included in the neural network device 10 can store the feature map data D_FM in the data buffer 141 in units of cells. Each cell can contain at least one feature.

[0137] In operation S200, the neural network IC 100 generates first data D1 by extracting feature data of one of the features contained in a plurality of cells included in the feature map data D_FM, each feature corresponding to the same coordinate value. For example, the first multiplexing circuit 143 included in the multiplexing circuit 142 can use at least one multiplexer to extract feature data of one of the features contained in the plurality of cells that correspond to the same coordinate value.

[0138] In operation S300, the neural network IC 100 can generate the extracted data Data_ext by rearranging the rows and / or columns of the first data D1. For example, the second multiplexing circuit 144 included in the multiplexing circuit 142 can generate the extracted data Data_ext by rotating the first data D1 in the vertical and / or horizontal direction using at least one multiplexer. In operation S400, the neural network IC 100 can perform neural network operations using the extracted data Data_ext. For example, the arithmetic circuit 160 can perform convolution by multiplying the extracted data Data_ext by the weight value corresponding to the extracted data Data_ext and adding the results of the multiplication. Figure 12 The feature map data D_FM and the extracted data Data_ext are shown to explain the data extraction process according to a conceptual embodiment of the present invention. Specifically, Figure 12 The data points are shown to illustrate an example where the window size required for the calculations is smaller than the size of each cell. For ease of explanation, Figure 12 The illustration shows an example where the feature map data D_FM contains 16 cells, each containing 16 features, and the window size required for computation is 3×2. However, the number of cells, the number of features, and the window size are merely examples. Therefore, embodiments of the present invention are not limited thereto. Reference will now be made to... Figure 5 and Figure 6 To explain

[0139] Figure 12 .

[0140] When the neural network IC 100 performs neural network operations (e.g., convolution) based on the data feature map, the arithmetic circuit 160 may require the extracted data Data_ext from the feature map data D_FM. In this case, the data extraction circuit 140 may extract the extracted data Data_ext from the feature map data D_FM.

[0141] The feature map data D_FM may contain cells 1 through 16 (Cell_16). The feature map data D_FM may be stored in data buffer 141 as cells 1 through 16. A portion of the extracted data Data_ext, necessary for neural network operations, is contained in cell 1, another portion in cell 2, another portion in cell 5, and yet another portion in cell 6. Reference will now be made to… Figures 14A to 14C To explain from Figure 12 Feature map data D_FM extraction shown Figure 12 The process of extracting the Data_ext shown.

[0142] Figures 13A to 13C This is a block diagram of multiplexing circuits 142a, 142b and 142c according to embodiments of the present invention.

[0143] Reference Figure 13A The multiplexing circuit 142a may include a first multiplexing circuit 143a, a second multiplexing circuit 144a, and / or a data selection circuit 147a. The first multiplexing circuit 143a and the second multiplexing circuit 144a may be connected to... Figure 6 The first multiplexing circuit 143 and the second multiplexing circuit 144 shown are substantially the same. The first multiplexing circuit 143a can generate first data D1 by extracting data from the feature map data D_FM. The data selection circuit 147a can generate selected first data D1_SEL by selecting only the data necessary for calculation, excluding data not required for calculation, from the first data D1. The second multiplexing circuit 144a can generate the extracted data Data_ext by rotating the selected first data D1_SEL in the vertical and / or horizontal directions. Figure 14A The middle shows with Figure 13A The data corresponding to the illustrated embodiment.

[0144] Reference Figure 13BThe multiplexing circuit 142b may include a first multiplexing circuit 143b and / or a second multiplexing circuit 144b, and the second multiplexing circuit 144b may include a vertical rotation circuit 145b, a horizontal rotation circuit 146b, a first data selection circuit 147b, and / or a second data selection circuit 148b. The first multiplexing circuit 143b, the vertical rotation circuit 145b, and the horizontal rotation circuit 146b may be substantially the same as the first multiplexing circuit 143, the vertical rotation circuit 145, and the horizontal rotation circuit 146 described above with reference to the foregoing figures. The first multiplexing circuit 143a can generate first data D1 by extracting data from the feature map data D_FM. The vertical rotation circuit 145b can generate first data D1_VR after vertical rotation by rotating the first data D1 in the vertical direction. The first data selection circuit 147b generates selected first data D1_SEL by selecting only the rows necessary for calculation, excluding those not required for the calculation, from the vertically rotated first data D1_VR. The horizontal rotation circuit 146b generates horizontally rotated first data D1_HR by rotating the selected first data D1_SEL horizontally. The second data selection circuit 148b generates extracted data Data_ext by selecting only the columns necessary for calculation, excluding those not required for the calculation, from the horizontally rotated first data D1_HR. Figure 14B The middle shows with Figure 13B The data corresponding to the illustrated embodiment.

[0145] Reference Figure 13C The multiplexing circuit 142c may include a first multiplexing circuit 143c, a second multiplexing circuit 144c, and / or a data selection circuit 147c. The first multiplexing circuit 143c and the second multiplexing circuit 144c may be substantially the same as the first multiplexing circuit 143 and the second multiplexing circuit 144 described above with reference to the foregoing figures. The first multiplexing circuit 143c can generate first data D1 by extracting data from the feature map data D_FM. The second multiplexing circuit 144c can generate rotated first data D1_R by rotating the first data D1 in a vertical and / or horizontal direction. The data selection circuit 147c can generate extracted data Data_ext by selecting only the data necessary for calculation, excluding data not required for calculation, from the rotated first data D1_R. Figure 14C The middle shows with Figure 13C The data corresponding to the illustrated embodiment.

[0146] Figures 14A to 14C The data is presented in matrix form to explain the various data points according to the conceptual embodiments of the present invention. Figures 13A to 13CThe data selection and extraction methods performed by the multiplexing circuits 142a, 142b and 142c shown.

[0147] Now refer to Figure 13A To explain Figure 14A The first data D1 generated by the first multiplexing circuit 142a may contain data lines (not shaded) that are not necessary for calculation. The data selection circuit 147a can generate the extracted data Data_ext by selecting only the data lines D11, D14, D31, D34, D41, and D44 necessary for calculation, excluding the data lines D12, D13, D21, D22, D23, D24, D32, D33, D42, and D43 that are not necessary for calculation, from the first data D1.

[0148] Now refer to Figure 13B To explain Figure 14B The first data D1 generated by the first multiplexing circuit 142b may contain data rows (not shaded) that are not required for calculation. The vertical rotation circuit 145b generates vertically rotated first data D1_VR by rotating the first data D1 vertically. The first data selection circuit 147b generates selected first data D1_SEL by excluding the first row (not required for calculation) from the vertically rotated first data D1_VR. The horizontal rotation circuit 146b generates horizontally rotated first data D1_HR by rotating the selected first data D1_SEL horizontally. The second data selection circuit 148b generates extracted data Data_ext by excluding the second and third columns (not required for calculation) from the horizontally rotated first data D1_HR. Reference will now be made to... Figure 13C To explain Figure 14C The first data D1 generated by the first multiplexing circuit 142c may include data lines (not shaded) that are not necessary for calculation. The second multiplexing circuit 143c can generate rotated first data D1_R by rotating the first data D1 in the vertical and / or horizontal direction. The data selection circuit 147c can generate extracted data Data_ext by selecting only the data lines D11, D14, D31, D34, D41, and D44 necessary for calculation, excluding the data lines D12, D13, D21, D22, D23, D24, D32, D33, D42, and D43 that are not necessary for calculation, from the rotated first data D1_R.

[0149] The inventive concept has been specifically illustrated and described with reference to exemplary embodiments thereof. The terminology used herein is for illustrative purposes only and is not intended to limit the inventive concept. Therefore, it should be understood that various changes in form and detail may be made to the described embodiments without departing from the spirit and scope of the inventive concept as defined by the appended claims.

Claims

1. An integrated circuit included in a means for performing neural network operations, the integrated circuit comprising: A buffer is configured to store feature map data in units of cells, each containing at least one feature, each of the at least one feature in each cell corresponding to a coordinate value, wherein the feature map data is used in the neural network operation; and A multiplexing circuit, including at least one multiplexer, is configured to: Receive the feature map data containing multiple cells from the buffer. The at least one multiplexer is configured to extract feature data of one feature selected from features corresponding to the same coordinate value in each of the plurality of cells, wherein the feature corresponding to the same coordinate value is a feature located at the same position in each of the plurality of cells. And output the extracted feature data.

2. The integrated circuit of claim 1, wherein the multiplexing circuit is configured to generate the first data by extracting feature data corresponding to all coordinate values ​​of the first data from the plurality of cells, and to generate the extracted feature data by rotating the first data along at least one of the vertical and horizontal directions.

3. The integrated circuit of claim 2, wherein the multiplexing circuit rotates the first data in the vertical direction by changing the order of the rows of the first data in matrix form, or rotates the first data in the horizontal direction by changing the order of the columns of the first data.

4. The integrated circuit according to claim 1, wherein the multiplexing circuit comprises: A first multiplexing circuit includes at least one multiplexer, the multiplexer being configured to extract feature data of a selected feature from the features corresponding to the same coordinate value in each of the plurality of cells and to output first data based on the extracted feature data. as well as The second multiplexing circuit includes multiple multiplexers, which are used to receive the first data from the first multiplexing circuit and rotate the received first data in a vertical or horizontal direction.

5. The integrated circuit of claim 4, wherein the first multiplexing circuit comprises a plurality of multiplexers configured to select one input signal from a number of input signals corresponding to the number of the plurality of cells contained in the feature map data.

6. The integrated circuit of claim 5, wherein the first multiplexing circuit comprises multiplexers in number corresponding to the number of the at least one feature contained in each cell.

7. The integrated circuit of claim 4, wherein the second multiplexing circuit comprises: A vertical rotation circuit is configured to rotate the first data in the vertical direction by changing the order of the rows of the first data in matrix form, and to generate first data after vertical rotation. as well as A horizontal rotation circuit is configured to rotate the vertically rotated first data along the horizontal direction by changing the order of the columns of the vertically rotated first data, and to generate the extracted feature data.

8. The integrated circuit according to claim 7, wherein The vertical rotation circuit includes multiplexers in a number corresponding to the number of rows of the first data, and The horizontal rotation circuit includes multiplexers in a number corresponding to the number of columns of the first data.

9. The integrated circuit according to claim 8, wherein The multiplexer included in the vertical rotation circuit is configured to select different rows from the rows of the first data, and The multiplexer included in the horizontal rotation circuit is configured to select different columns from the columns of the first data.

10. The integrated circuit of claim 4, wherein the second multiplexing circuit comprises: A horizontal rotation circuit is configured to rotate the first data in the horizontal direction by changing the order of the columns of the first data in matrix form, and to generate first data after horizontal rotation. as well as A vertical rotation circuit is configured to rotate the horizontally rotated first data along the vertical direction by changing the order of the rows of the horizontally rotated first data, and to generate the extracted feature data.

11. The integrated circuit according to claim 1, wherein The neural network operations include convolution operations. The feature map data includes the input feature map data to which the convolution operation will be performed, and The extracted feature data is the data that will be directly multiplied by weights during the convolution operation.

12. A neural network processor configured to perform neural network operations, the neural network processor comprising: An internal memory is configured to receive external data from outside the neural network processor and store data and / or weights based on the external data; A data extraction circuit is configured to receive data from the internal memory, store the received data as feature map data, the feature map data being divided into multiple cells, each cell including at least one feature, wherein the at least one feature of each cell corresponds to a coordinate value, classify multiple features in the multiple cells contained in the feature map data into multiple groups, wherein each group of the multiple groups includes features in the multiple cells corresponding to the same coordinate value, wherein the feature corresponding to the same coordinate value is a feature located at the same position in each of the multiple cells, and output the extracted data by extracting feature data of one feature from the features of each group of the multiple groups; as well as An arithmetic circuit is configured to perform the neural network operation using the extracted data provided by the data extraction circuit.

13. The neural network processor of claim 12, wherein the arithmetic circuitry is further configured to perform a convolution operation on the feature map data by multiplying the extracted data by the weights provided by the internal memory.

14. The neural network processor of claim 12, wherein the data extraction circuit comprises: A buffer is configured to store the received data as the feature map data in units of cells; as well as The multiplexing circuit is configured to output the extracted data by extracting the data necessary for the neural network operation from the feature map data received from the buffer.

15. The neural network processor of claim 14, wherein the multiplexing circuit includes a first multiplexing circuit, the first multiplexing circuit including a plurality of multiplexers corresponding to the plurality of groups respectively, each of the plurality of multiplexers being configured to extract feature data of a feature from each of the plurality of groups.

16. The neural network processor of claim 15, wherein... The first multiplexing circuit is configured to output first data based on the feature data extracted by the plurality of multiplexers, and The multiplexing circuit further includes a second multiplexing circuit configured to generate the extracted data by rearranging the first data received from the first multiplexing circuit along a horizontal and / or vertical direction.

17. The neural network processor of claim 15, wherein... The first multiplexing circuit is configured to output first data based on the feature data extracted by the plurality of multiplexers, and The multiplexing circuit further includes: A vertical rotation circuit is configured to change the order of rows of the first data in matrix form and includes multiplexers of a number corresponding to the number of rows of the first data. as well as A horizontal rotation circuit is configured to change the order of the columns of the first data and includes multiplexers in a number corresponding to the number of columns of the first data.

18. The neural network processor of claim 12, wherein the size of the extracted data is equal to or less than the size of the cell.

19. A data processing method executed by a neural network processor configured to perform neural network operations, the data processing method comprising: Feature map data for neural network operations is stored in units of cells, each containing at least one feature, wherein each of the at least one feature in each cell corresponds to a coordinate value; and First data in matrix form is generated by extracting feature data of one feature corresponding to the same coordinate value from each of the multiple cells, wherein the feature corresponding to the same coordinate value is a feature located at the same position in each of the multiple cells included in the feature map data. Extracted data for use in the neural network operation is generated by rearranging the rows and / or columns of the first data; and The neural network operation is performed using the extracted data, wherein the execution of the neural network operation is performed by an arithmetic circuit.

20. The data processing method of claim 19, wherein generating the first data comprises generating the first data by extracting the feature data using a plurality of multiplexers corresponding to the plurality of coordinate values.

21. The data processing method according to claim 19, wherein generating the extracted data comprises: The first data is rotated vertically by changing the order of the rows of the first data using multiple multiplexers. as well as The extracted data is generated by rotating the vertically rotated first data horizontally by changing the order of the columns of the first data using the plurality of multiplexers.

22. The data processing method of claim 19, wherein generating the first data includes extracting feature data of a feature selected from the features corresponding to the same coordinate value in each of the plurality of cells based on the position of the extracted data necessary for the neural network operation on the feature map data.

23. A neural network device configured to perform neural network operations, the neural network device comprising: At least one processing unit; as well as A neural network processor is configured to communicate with the at least one processing unit via a system bus and output information signals by performing neural network operations, including convolution operations, based on input data provided by the at least one processing unit. The neural network processor is configured to divide the input data into multiple cells, each cell containing at least one feature, each of the at least one feature in each cell corresponding to a coordinate value, and to store the input data as an input feature map. Feature data of one feature selected from features corresponding to the same coordinate value in each of the plurality of cells, wherein the feature corresponding to the same coordinate value is a feature located at the same position in each of the plurality of cells, and the extracted data is generated for use in the convolution operation.

24. The neural network apparatus of claim 23, wherein the neural network processor is configured to generate first data by extracting the feature data and to generate the extracted data by rotating the first data in a horizontal and / or vertical direction.

25. An integrated circuit included in an apparatus for performing neural network operations using extracted data extracted from feature map data, the integrated circuit comprising: The buffer is configured to store the feature map data in units of cells, each containing at least one feature; A first multiplexing circuit is configured to extract first data from the feature map data using a number of multiplexers less than or equal to the number of the at least one feature contained in each cell, wherein the first data contains all the feature data contained in the extracted data. as well as The second multiplexing circuit is configured to generate the extracted data by rearranging the rows and / or columns of the first data using a multiplexer whose number is less than or equal to the sum of the number of rows and the number of columns of the first data.

Images