Artificial intelligence acceleration hardware device and operating method thereof

Abstract

The present application relates to a real-time artificial intelligence acceleration hardware device with high efficiency for memory access and an operating method thereof. The device includes a neural network unit connected according to a neural network structure to implement an operation of a neural network, a controller for processing data exchange with an off-chip memory and controlling an operation of the neural network unit, and an on-chip memory including a row buffer for providing an input of the neural network, a weight buffer for storing weight values used during an operation process of the neural network unit, and an output buffer for storing an output of the neural network.

Description

Technical Field

[0001] This disclosure relates to artificial intelligence acceleration hardware technology, and more specifically, to an artificial intelligence acceleration hardware device and its operating method, which implements the U-net structure as a pipeline unit and minimizes access to off-chip memory through row-by-row neural network operations. Background Technology

[0002] To achieve high performance and low power consumption, modern CNN hardware has been developed. In particular, the increased access to off-chip memory presents various challenges and limitations. For example, in terms of latency, access to off-chip memory is much slower than access to on-chip memory, thus significantly reducing the overall system processing speed.

[0003] Furthermore, accessing off-chip memory consumes more energy than accessing on-chip memory. Meanwhile, the memory controller, which controls access to off-chip memory, performs tasks such as processing memory access requests and managing data. In controlling the memory controller, various tasks such as address management, data transfer, and error detection and correction are performed, increasing the complexity of memory management.

[0004] Therefore, traditional CNN hardware can have fundamental design principles that effectively support parallel processing, optimized memory hierarchies, and data reuse to achieve high throughput and low power consumption. However, to achieve real-time performance requiring significant computation and memory access, such as noise removal from camera images, an optimized hardware architecture design may be necessary.

[0005] Furthermore, camera images may inherently contain noise, particularly from fog, sea fog, and yellow dust, with dust levels potentially becoming very severe depending on weather conditions. Addressing this issue might require real-time, ultra-high-speed noise removal techniques based on artificial intelligence. However, real-time processing is likely very challenging with existing acceleration technologies due to the enormous computational and memory access required.

[0006] U-net can be considered a representative neural network that exhibits excellent performance in noise removal. U-net can include: an encoding part, which includes CNN computation; a decoding part; and skip connections between the encoding and decoding parts that send the encoded results to the decoding part. This construction preserves detailed image information, thus enabling the recovery of images damaged by noise.

[0007] Korean Patent Registration No. 10-2023-0117953 (August 10, 2023) discloses a noise removal device and a noise removal method using the noise removal device. Summary of the Invention

[0008] Technical issues

[0009] One embodiment of this disclosure aims to provide an artificial intelligence acceleration hardware device and its operation method for noise removal, which minimizes access to off-chip memory through row-by-row neural network computation by implementing a representative neural network U-net structure, which exhibits excellent performance in noise removal, as a pipeline unit.

[0010] One embodiment of this disclosure aims to provide an efficient real-time artificial intelligence acceleration hardware device and its operation method, which minimizes access to off-chip memory during real-time noise removal by directly connecting units that process neural network calculations row by row based on a pipelined neural network hardware structure to SRAM.

[0011] means for solving problems

[0012] In one embodiment, an artificial intelligence acceleration hardware device includes: a neural network unit (unit) connected according to a neural network structure to implement the operation of the neural network; a controller that processes data exchange with off-chip memory and controls the operation of the neural network unit; and an on-chip memory including a row buffer (row buffer) for providing inputs to the neural network, a weight buffer (weight buffer) for storing weight values ​​used during the operation of the neural network unit, and an output buffer (output buffer) for storing the outputs of the neural network.

[0013] The neural network unit may include: an encoding unit that processes encoding to compress input data into encoded data in a neural network with an encoder-decoder structure; a decoding unit that processes to recover the encoded data; and a bottleneck unit that connects the encoding unit and the decoding unit. Each neural network unit may include: a shifter module that performs a shift operation on the input data; a weight register module that receives weight values ​​from a weight buffer (WeightBuf) during convolution computation; a temporary buffer module that stores intermediate feature maps generated during the convolution computation process; and a convolutional unit module that implements at least one convolutional layer to perform the convolution computation.

[0014] Each coding unit may also include a pool buffer module, which downsamples the convolution calculation results performed by the convolution unit module.

[0015] The convolutional unit module may include: at least one PE module comprising a plurality of multipliers; at least one adder summing the multiplication results of the plurality of multipliers; and at least one Tsum module storing the summation result calculated by the adder. The convolutional unit module can be implemented by variably applying the number of multipliers, the number of PE modules, the number of adders, and the number of Tsum modules according to at least one of the number of input channels, the number of output channels, and the filter size set in the convolutional layer.

[0016] The controller can access off-chip memory before the start phase of the neural network operation to read input data and weight values ​​for the operation of the neural network, and can execute control to store output data based on the operation of the neural network by accessing off-chip memory after the end phase of the neural network operation.

[0017] The controller can perform control to start filling the buffer data when the buffer data stored in the row buffer overlaps with the edge region of the input data.

[0018] The on-chip memory may also include a cascading buffer (CatBuf) that connects to the neural network units and stores cascaded feature maps generated during the operation of the neural network units. Row buffers can store input data read from off-chip memory by dividing the input data into rows, and can provide input data to the neural network units row by row according to control signals from the controller. Weight buffers can be directly connected to the neural network units and can store weight values ​​used during the operation of the neural network units. Cascading buffers can directly connect the encoding and decoding units within the neural network units, and can be connected sequentially in a number independently calculated based on the depth of the neural network.

[0019] In one embodiment, an operation method performed in an artificial intelligence acceleration hardware device comprising neural network units (units) connected according to a neural network structure includes the following steps: retrieving weight values ​​used during the operation of the neural network units from off-chip memory and storing the retrieved weight values ​​in a weight buffer (WeightBuff) independently connected to the neural network units; providing the neural network input data read from the off-chip memory to the neural network units on a row-by-row basis via a row buffer (row buffer); sequentially performing convolution calculations for the operation of the neural network via the neural network units; storing the result of the convolution calculations as the output of the neural network via an output buffer (output buffer); and retrieving the output of the neural network from the output buffer and storing the output in the off-chip memory.

[0020] The steps of performing convolution calculations sequentially may include: padding the buffer data when the buffer data provided by the row buffer overlaps with the edge regions of the input data.

[0021] In an implementation, an operation method performed in an artificial intelligence acceleration hardware device comprising neural network units (units) connected according to a neural network structure includes the following steps: providing neural network inputs retrieved from off-chip memory to the neural network unit on a row-by-row basis via a row buffer (row buffer); performing convolution calculations for encoding processing during operation of the neural network via an encoding unit in the neural network unit; storing concatenated feature maps generated by the encoding unit via a concatenation buffer (CatBuf); receiving the concatenated feature maps from the concatenation buffer via a decoding unit in the neural network unit; performing convolution calculations for decoding processing during operation of the neural network; storing the result of the decoding processing as the output of the neural network via an output buffer (output buffer); and retrieving the output of the neural network from the output buffer and storing the output in off-chip memory.

[0022] Beneficial effects of the present invention

[0023] The disclosed technology can have the following effects. However, this does not mean that a particular implementation must include all of the following effects, or must include only the following effects, and the scope of the rights of the disclosed technology should not be construed as limited thereto.

[0024] According to one embodiment of the present disclosure, a real-time artificial intelligence acceleration hardware device and its operation method for noise removal can be implemented as a pipeline unit using a representative neural network U-net structure that will exhibit excellent performance in noise removal, minimizing access to off-chip memory through row-by-row neural network computation.

[0025] According to one embodiment of this disclosure, a real-time artificial intelligence acceleration hardware device and its operation method that are efficient in terms of memory access can minimize access to off-chip memory during real-time noise removal by directly connecting the individual units that process neural network calculations row by row based on a pipelined neural network hardware structure to SRAM. Attached Figure Description

[0026] Figure 1 This is a diagram illustrating one implementation of CNN convolution computation.

[0027] Figure 2 This is a diagram illustrating the U-net structure.

[0028] Figure 3 This is a diagram illustrating the hardware architecture for accelerating CNNs.

[0029] Figure 4 This is a diagram illustrating a weighted static data flow.

[0030] Figure 5 This is a diagram illustrating a neural network hardware structure according to the present disclosure.

[0031] Figure 6 This is an example Figure 5 A diagram showing the operational processes in each neural network unit.

[0032] Figure 7 This is an example Figure 5 A diagram showing the block diagrams of each neural network unit.

[0033] Figure 8 This is a block diagram illustrating the hardware structure of the coding unit according to the present disclosure.

[0034] Figure 9 This is a block diagram illustrating the hardware structure of the decoding unit according to the present invention.

[0035] Figure 10 This is a diagram illustrating the basic structure of a convolutional unit module according to this disclosure.

[0036] Figure 11 and Figure 12 The diagram illustrates various implementations of the convolutional unit module according to this disclosure.

[0037] Figure 13 This is a diagram illustrating the filling process according to this disclosure.

[0038] Figure 14 This is a diagram illustrating the filling process for line-based input according to this disclosure.

[0039] Figure 15 This is a diagram illustrating the skip connection structure of a serial buffer according to the present disclosure. Detailed Implementation

[0040] The description in this disclosure is merely an example of an implementation for structural or functional explanation, and the scope of this disclosure should not be construed as limited to the implementations described in the specification. That is, since implementations can be modified in various ways and can have various forms, the scope of this disclosure should be understood to include equivalents capable of realizing the technical concept. Furthermore, the purposes or effects presented in this disclosure do not imply that a particular implementation must include all or only such purposes or effects, and therefore the scope of this disclosure should not be construed as limited thereto.

[0041] Meanwhile, the meanings of the terms described in this specification should be understood as follows.

[0042] The use of terms such as "first" and "second" is to distinguish one component from another, and the scope of rights should not be limited by these terms. For example, a first component can be named a second component, and similarly, a second component can be named a first component.

[0043] When describing a component as "connected" to another component, the component can be directly connected to the other component, but the other component can exist between them. Conversely, when describing a component as "directly connected" to another component, it should be understood that no other component exists in between. Similarly, other expressions describing relationships between components, such as "between" and "directly between," or "adjacent" and "directly adjacent," should be interpreted in the same way.

[0044] Singular expressions should be understood to include plural expressions unless otherwise expressly indicated in the context, and terms such as “including” or “comprising” should be understood to specify the presence of the stated feature, number, step, operation, component, part or combination thereof, but should not be understood to exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0045] For ease of description, identification symbols (e.g., a, b, c, etc.) are used in each step. These symbols do not describe the order of each step, and each step may occur in a different order than described unless the specific order is clearly described in the context. That is, each step may occur in the same order as described, may be performed substantially simultaneously, or may be performed in the reverse order.

[0046] This disclosure can be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices that store data readable by a computer system therein. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage devices, etc. Furthermore, the computer-readable recording medium can be distributed across a network-connected computer system, allowing the computer-readable code to be stored and executed in a distributed manner.

[0047] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted as having an ideal or overly formal meaning, unless expressly defined in this specification.

[0048] Figure 1 This is a diagram illustrating one implementation of CNN convolution computation.

[0049] Reference Figure 1 CNN (Convolutional Neural Network) can be considered the most widely used neural network in machine learning tasks related to image processing. CNNs can include operations such as convolution, ReLU (Rectified Linear Unit), and pooling, with convolution being the core component. That is, convolution can be understood as applying a filter (or kernel) as a small matrix to a portion of the input image, calculating matching element-wise products, and summing the results. In this case, the calculated values ​​can be transformed into pixel values ​​in a new image (or feature map).

[0050] Furthermore, while moving filters to other parts of the image, convolution operations can be repeatedly performed on the entire image. As a result, the generated feature maps can highlight and represent the unique features of the original image (e.g., edges, colors, textures, etc.). Additionally, filter values ​​can be generated during the CNN's learning process.

[0051] Specifically, in Figure 1 In the diagrams (a), (b), and (c), we can represent the input, the filter, and the result of the convolution calculation, respectively. For example, the original input image can correspond to the grid region in diagram (a), and diagram (b) can show the filter applied to the shaded region. Meanwhile, the shaded region in diagram (c) can correspond to the result of the convolution calculation for the shaded regions in diagrams (a) and (b). While the filter moves across the input image, a 2D convolution operation can be performed by calculating the sum of element-wise multiplications of pixels at the same location.

[0052] Figure 2 This is a diagram illustrating the structure of the U-net neural network.

[0053] Reference Figure 2 U-net can be mainly composed of two parts. One part can correspond to the "shrinking path" or "encoding path" that extracts features from the image, while the other part can correspond to the "expanding path" or "decoding path" that generates an output with the same resolution as the original image based on the extracted features.

[0054] The encoder can perform downsampling for the purpose of capturing context and can include multiple convolutional layers and max-pooling layers similar to a typical CNN. Figure 2 In this system, the encoder can consist of four coding blocks (En 0, En 1, ..., En 3), and a coding block can consist of three CNN layers and a downsampling layer.

[0055] Furthermore, the decoder can perform upsampling for localization purposes and can include multiple upsampling layers and CNN layers symmetrical to the encoder. Figure 2 In this context, the decoder can consist of four decoding blocks (De 0, De 1, ..., De 3), and a decoding block can consist of three CNN layers and an upsampling layer.

[0056] Skip connections can link symmetrically related encoded and decoded blocks. This connection allows coarse information from the shrinking path to be integrated into the decoding process, thereby improving learning efficiency.

[0057] The bottleneck can consist of the lowest-dimensional layer in a neural network. That is, the bottleneck can be composed of the CNN layers between the final downsampling and the first upsampling. Therefore, the bottleneck can have features including high-level features.

[0058] The U-net architecture can preserve fine-grained features (detailed and detailed information or structure) of images while obtaining high-level features through deep networks, thus achieving detailed predictions at the pixel level. U-net's strong feature extraction capabilities and high-resolution output generation capabilities can be very advantageous for recovering the original image from noisy images.

[0059] Figure 3 This is a diagram illustrating the hardware architecture for accelerating CNNs.

[0060] Reference Figure 3 CNN acceleration hardware architecture can include a hierarchical structure of off-chip memory, on-chip memory, and processing elements (PEs). Off-chip memory can correspond to the memory with the largest capacity in the main memory and can be mainly used to store model parameters (weights) and input data.

[0061] On-chip memory can correspond to memory with faster access speeds than off-chip memory but smaller capacity. This memory can primarily be used to store intermediate feature maps generated during the CNN operation. Processing elements can correspond to the computational resources that actually perform the CNN operations.

[0062] The operation of CNN acceleration hardware can include a data loading process from off-chip memory, a data distribution process from on-chip memory, an operation execution process within processing elements, and a result storage process in on-chip memory. Specifically, the data loading process can correspond to loading the input data and weight values ​​required to begin CNN computation from off-chip memory. Afterward, the data can be transferred to on-chip memory for fast access. Furthermore, the data distribution process can correspond to storing the loaded data in on-chip memory, known as a global buffer, and then distributing it from the global buffer to individual processing elements (PEs) as needed.

[0063] Furthermore, the operation execution process can correspond to each PE performing the required computations using given data and parameters (weights), then storing the computation results back in on-chip memory, and using the stored computation results as input for the next computation stage. Finally, the result loading process can correspond to loading the final result from on-chip memory after the CNN computation is complete.

[0064] In addition, Figure 3 In 2D PE array schemes, tiling techniques are primarily used to implement CNNs. Because CNNs need to process large amounts of data (e.g., input feature maps, filters, output feature maps), and the on-chip memory of accelerator hardware has a limited size, the entire dataset cannot be stored simultaneously. To address this issue, processing can be performed on the data after it has been divided into small "tiles." Each tile can be tailored to the size of the 2D PE array, and only one tile can be loaded into the on-chip memory at a time.

[0065] Using off-chip memory to store or load data for which tiled techniques are applied can be inefficient in terms of hardware speed and power consumption. To address this issue, various data reuse strategies can be used. These strategies can include: a weighted static (WS) method that stores and reuses filter weights (weights) in a PE register file; an output static (OS) method that reuses parts and psums; and a row static (RS) method that reuses filters, input feature maps, and psums.

[0066] Figure 4 This is a diagram illustrating a weighted static data flow.

[0067] Reference Figure 4 Weighted static (WS) dataflow technique corresponds to one of the dataflow strategies used in the implementation of CNN hardware, and can be performed in a way that moves other data while the weight data is fixed.

[0068] First, weights can be loaded into each PE, and these weights can be fixed until the corresponding input operation is completed. Figure 4 In this context, such weights can be represented as W0, W1, W2, and W3. The input feature map can be streamed to the PE array as... Figure 4 The activation of this could mean that sequential activations are used in operations within the PE.

[0069] Each PE can calculate a partial sum Psum and can read the previous partial sum Psum n-1 from on-chip memory and cumulatively add it to the other Psums. The cumulative partial sum Psum n can be sent back to on-chip memory. When the activation operation for the weights of a set is completed, the weights of the next set can be loaded into the PE and fixed, and the same process can be repeated.

[0070] The main feature of the WS method is that the weight data is fixed in the local memory of the PE and can be reused along with the input data (reusability). However, it may have the following drawbacks: the movement of input data and intermediate feature maps may occur frequently, while the weight data is fixed.

[0071] Figure 5 This is a diagram illustrating a neural network hardware structure according to the present disclosure.

[0072] Reference Figure 5 The real-time artificial intelligence acceleration hardware device 500 can be implemented as a neural network hardware structure for real-time noise removal operations, and through this neural network hardware structure, access to DRAM corresponding to off-chip memory can be minimized. For this purpose, the real-time artificial intelligence acceleration hardware device 500 may include design decision elements such as a "pipeline neural network structure," "direct connections between cells and SRAM," "buffers for skipping connections," and "row-by-row neural network computation."

[0073] In one embodiment, the real-time AI acceleration hardware device 500 can be implemented to perform operations in conjunction with a user terminal. Here, the user terminal may correspond to a terminal device operated by a user. In particular, the user terminal may include service modules utilizing various AI computations. That is, the user terminal may correspond to a device that constructs an AI model by executing the AI ​​acceleration hardware method according to this disclosure and utilizes the constructed model. For example, the user terminal may capture a camera image and request the construction of an AI model for removing noise included in the camera image.

[0074] Furthermore, the user terminal can be implemented as a smartphone, laptop computer, or computer capable of operating by connecting to the real-time AI acceleration hardware device 500, and is not limited to these, and can be implemented as various devices including tablet PCs. In particular, the user terminal can install and execute dedicated programs or applications for interacting with the real-time AI acceleration hardware device 500. Additionally, the user terminal can operate by interacting with the real-time AI acceleration hardware device 500, which is a device constituting the real-time AI acceleration hardware system.

[0075] Furthermore, the real-time AI acceleration hardware device 500 can be implemented to operate by connecting to a separate external system. For example, the real-time AI acceleration hardware device 500 can operate by interacting with a preprocessing system that provides data preprocessing or an AI system that defines and manages AI models.

[0076] Specifically, the real-time artificial intelligence acceleration hardware device 500 can pipeline the various layers constituting the neural network using units such as encoding units 531, decoding units 535, and bottleneck units 533. Each unit can correspond to a neural network unit connected according to the neural network structure to implement the operation of the neural network, and can be adapted for parallel processing because each unit can perform independent computations. Furthermore, since the layers included in each unit are optimized according to the system specifications of the PE array, only the required number of PEs can be used, thereby reducing unnecessary resource usage.

[0077] Furthermore, the real-time AI acceleration hardware device 500 can be implemented as an on-chip memory directly connected to all parameters (i.e., weight values) used in each unit. Additionally, intermediate feature maps generated within each unit can also be stored in and loaded from a temporary buffer TempBuf directly connected to the convolutional unit (convolutional unit) module. Therefore, the real-time AI acceleration hardware device 500 can access off-chip memory only during the initial loading of image input and parameters (weights). On the other hand, in conventional methods, since all layers are accelerated sequentially and a fixed-size 2D PE array is used, layers with different specifications may suffer from increased complexity due to the application of tile techniques based on the size of the 2D PE array.

[0078] Furthermore, the real-time AI acceleration hardware device 500 can utilize only on-chip memory to store concatenated feature maps for skipping connections. Because on-chip memory offers fast access times and can process predetermined data units, precise control of the tiles is possible. Conversely, in conventional methods, concatenated feature maps are stored in DRAM and loaded when necessary, leading to increased power consumption and time latency.

[0079] Furthermore, to simplify padding, the real-time AI acceleration hardware device 500 can apply a row-based segmentation method instead of the traditional block-based segmentation method. Padding can correspond to operations processed at the edges, and when input is received in rows, the number of padding processing instances can be reduced compared to when input is received in blocks, thus simplifying the processing.

[0080] exist Figure 5 In this context, the real-time AI acceleration hardware device 500 can be implemented as a hardware structure in which no external access occurs except when loading the image input (input) and parameters (weights) of the neural network and outputting the final result. For example, the real-time AI acceleration hardware device 500 can be implemented as a unit that includes implementing a neural network with an encoder-decoder structure.

[0081] More specifically, the hardware structure may consist of an encoding unit (EN 0, EN 1) 531, a decoding unit (DE 0, DE 1) 535, a bottleneck unit (bottleneck) 533, an FC unit 537, and multiple buffers. Additionally, the hardware structure may include a controller (controller) 510 that handles data exchange with off-chip memory and controls the operation of each neural network unit. The encoding unit 531 can handle encoding processing to compress input data into encoded data, the decoding unit 535 can handle decoding processing to recover the encoded data, and the bottleneck unit 533 can connect the encoding unit 531 and the decoding unit 535.

[0082] In this configuration, buffers may include a row buffer (row buffer) 551 providing input to the neural network, a weight buffer (weight buffer) 553 storing weight values ​​used during the operation of neural network units, an output buffer (output buffer) 555 storing the output of the neural network, and a concatenated buffer CatBuf 557 for storing feature maps required for skip connections. Furthermore, these buffers can be implemented on-chip memory, allowing processing to be performed without accessing off-chip memory. For example, each buffer can be logically defined by a separate region partitioned on a single on-chip memory, or it can be implemented as multiple physically separate on-chip memories.

[0083] In one implementation, each row buffer 551 can store input data retrieved from off-chip memory by dividing the input data into rows, and can provide input data to the neural network unit in rows according to the control signals of the controller 510.

[0084] Additionally, the weight buffer 553 can be directly connected to each individual neural network unit. The weight buffer 553 can also be independently connected to each individual neural network unit and can store all weight values ​​required for the CNN operations performed during the operation of the neural network unit. That is, weight values ​​can be loaded from external memory (i.e., off-chip memory) along with the input data and then used to construct the neural network.

[0085] In one embodiment, a concatenation buffer CatBuf 557 for storing concatenation feature maps used for skipped connections can be arranged between the encoding unit 531 and the decoding unit 535. In this case, the number of concatenation buffers 557 can be determined according to the pipeline structure, and the number of concatenation buffers 557 optimized for the pipeline can play an important role in accurately controlling the timing of the neural network.

[0086] For example, for pipelined processing, three serial buffers CatBuf 557 may be needed between the EN 0 unit and the DE 0 unit. Conversely, one serial buffer 557 is sufficient between the EN 1 unit and the DE 1 unit. If additional encoding units 531 or decoding units 535 are added to have the same structure, two additional serial buffers 557 may be needed to match the timing of the pipelined processing.

[0087] In one embodiment, the controller 510 may access off-chip memory before the start phase of the neural network operation to retrieve input data and weight values ​​for the neural network operation, and may perform control such that output data based on the neural network operation is stored by accessing off-chip memory after the end phase of the neural network operation. In another embodiment, the controller 510 may perform control such that when buffer data stored in row buffer 551 in row units overlaps with the edge regions of the input data, a filling operation of the corresponding buffer data is initiated.

[0088] Figure 6 This is an example Figure 5 A diagram showing the operational processes in each neural network unit.

[0089] Reference Figure 6 Figure (a) can correspond to the operation process of the encoding unit (encoding unit), and Figures (b) and (c) can correspond to the operation processes of the bottleneck unit (bottleneck unit) and the decoding unit (decoding unit), respectively.

[0090] Specifically, all weights used in the neural network can be sent to the weight buffer (WeightBuff) connected to the corresponding unit. For example, weights used in the EN 0 unit can be sent to the weight buffer connected to the EN 0 unit. When all weights are loaded, the encoding unit can receive the feature map through the row buffer (Row Buffer). When the input feature map is sent, weights can be sent from the weight buffer to the encoding unit, and convolution operations can begin. In this case, the operation can be performed sequentially according to the layer order within the unit.

[0091] Additionally, when the operation result corresponds to a connected feature map, the result can be stored in a concatenation buffer (CatBuf). If negative, it can be determined whether the operation corresponds to the last layer, and if it is the last layer, the operation of the encoding unit can be terminated. If negative, new input can be received through the row buffer, and the same process can be repeated.

[0092] The bottleneck unit can receive the output of the last encoding unit as input. Once the input feature map has been sent, weights can be sent from the weight buffer to the bottleneck unit, and convolution operations can begin. This operation can be performed sequentially according to the layer order within the unit. It can be determined whether the operation corresponds to the last layer, and if so, the bottleneck unit's operation can terminate. If not, new input can be received, and the same process can be repeated.

[0093] The decoding unit can receive the output of the bottleneck unit as input. Since the decoding unit uses concatenated feature maps as input, it can receive these maps from the concatenation buffer. Once the input transmission is complete, weights can be sent from the weight buffer to the decoding unit, and convolution operations can begin. This operation can be performed sequentially according to the layer order within the unit. It can be determined whether the operation corresponds to the last layer, and if so, the decoding unit's operation can be terminated. If not, new input can be received, and the same process can be repeated.

[0094] Figure 7 This is an example Figure 5 A diagram showing the block diagrams of each neural network unit. Figure 8 This is a block diagram illustrating the hardware structure of the encoding unit according to the present disclosure, and Figure 9 This is a block diagram illustrating the hardware structure of the decoding unit according to the present disclosure.

[0095] Reference Figure 7Real-time artificial intelligence acceleration hardware devices may include neural network units (cells) connected according to a neural network structure to implement the operations of the neural network. In this case, the neural network unit may include an encoding unit (encoding unit), a bottleneck unit (bottleneck unit), a decoding unit (decoding unit), and a fully connected (FC) unit (FC unit). The units constituting the neural network may consist of modules based on convolution operations (convolution). Furthermore, the detailed configuration may vary slightly depending on the function of each unit. For example, depending on the specific function, different configurations may be used to implement units including downsampling, residual connections, or concatenation.

[0096] The basic structure typically used in each unit (the basic structure) can include shifters, convolutional units, weight buffers (WeightBuf), and temporary buffers (TempBuf) modules. Specifically, in the shifter module, shift operations can be performed on the input data for convolution calculations. In this case, 2D convolution operations can be performed by sliding a filter over the input feature map in the pixel unit.

[0097] Furthermore, the weight buffer module can receive and store all weights used in each unit from the host (host). The weight register module can receive the weights required for convolution computation from the weight buffer module. The temporary buffer module can store intermediate feature maps generated in layers other than the last convolutional layer of each unit. In the convolutional unit module, the actual convolution computations of the CNN layers included in each unit can be performed sequentially. The convolutional unit module can be configured to optimize PE according to the layer configuration.

[0098] Reference Figure 8 Each coding unit (CMU) can perform downsampling, and the convolutional unit module can generate concatenated feature maps. That is, each coding unit can include features for... Figure 7 The basic structure performs downsampling using a pool buffer module. That is, the pool buffer module can perform downsampling relative to the result of convolution computation performed by the convolution unit module. In one implementation, each coding unit can be directly connected to a concatenated buffer (CatBuf) that stores the concatenated feature maps.

[0099] Reference Figure 9 Each decoding unit (DAC) can be connected to a concatenation buffer (CatBuf) that stores the concatenated feature maps to receive them. Meanwhile, bottleneck units (BIX) and full-joint (FC) units can consist of only convolutional computations and can therefore be used with... Figure 7 It is implemented using the same basic structure.

[0100] Figure 10This is a diagram illustrating the basic structure of a convolutional unit module according to this disclosure. Figure 11 and Figure 12 The diagram illustrates various implementations of the convolutional unit module according to this disclosure.

[0101] Reference Figure 10 Each unit can be implemented as including a convolutional unit module for convolutional computation. The convolutional unit module can include PEs optimized according to the detailed specifications of the convolutional layer. In traditional approaches using fixed-size 2D PE arrays, tiling may be necessary for layers that could increase complexity, leading to wasted resources, and some PEs may remain inactive depending on the characteristics of the individual layers.

[0102] In one implementation, a convolutional unit module may consist of multiple processing elements (PEs), adders, and a Tsum representing the result of the convolution computation. Specifically, each PE may consist of a group of multipliers comprising multiple multipliers, rather than a MAC unit included in typical CNN hardware. Multiplication between weights and activations can be performed in each PE. Adders can receive the results from each PE and perform a summation operation, and the result can be stored in Tsum. That is, adders can be connected to PEs to receive the results of multiple multipliers included in a PE and perform a summation operation. The Tsum storing the computation results can be connected to a temporary buffer or an output buffer depending on the position of the corresponding computation in the neural network structure. If Tsum corresponds to an intermediate feature map of the layer currently being computed, Tsum can be stored in a temporary buffer (TempBuf), and if Tsum corresponds to a final feature map, Tsum can be stored in an output buffer (output buffer).

[0103] [Table 1]

[0104] In the case of Table 1 above, it can correspond to one implementation of the layer configuration in a specific unit. In this implementation, the configuration can consist of two consecutive CNN layers, and filter sizes of 3×3 and 1×1 can be used. Meanwhile, it is obvious that filters of various sizes can be applied. (Refer to...) Figure 11 This can correspond to one implementation of layer 1 in Table 1, where the convolutional unit module processes data. This convolutional unit module can use a total of 30 PEs, each consisting of 9 multipliers. That is, in layer 1, a 3×3 filter can be used, and since the number of input channels is 10, 90 multipliers might be needed to compute one output channel. Therefore, one output channel can be processed using 10 PEs. Figure 11In this case, since 30 PEs can be used, three channels can be output simultaneously (i.e., Tsum 0, 1, and 2). Layer 1 can include a total of 45 output channels, and the convolution calculation for Layer 1 can be completed when the operation is repeated 15 times. The resulting values ​​(i.e., Tsum 0, 1, 2, …, 44) can be used as input to Layer 2 and stored in a temporary buffer (TempBuf) for this purpose. See reference. Figure 12 This can correspond to one implementation of layer 2 in the convolutional unit module processing Table 1. The input (activation) to layer 2 can be sent from the temporary buffer (TempBuf) (i.e., the final feature map of layer 1). In layer 2, a 1×1 filter can be used, and since the number of input channels is 45, 45 multipliers may be needed to compute one output channel. Therefore, one output channel can be processed by 5 PEs. Figure 12 In this case, since 30 PEs can be used, six channels can be output simultaneously (i.e., Tsum 0, 1, …, 5). Layer 2 can include a total of 60 output channels, and the convolution calculation of Layer 2 can be completed when the operation is repeated 10 times. The resulting value (i.e., Tsum 0, 1, 2, …, 59) can be sent to the output buffer (output buffer). In one implementation, the number of multipliers, the number of PE modules, the number of adders, and the number of Tsum modules can be flexibly adjusted according to at least one of the number of input channels, the number of output channels, and the filter size set in the convolutional layer. The convolutional unit modules can be variably applied according to the configuration and detailed specifications of the layers included in the neural network hardware. Therefore, the configuration elements of the convolutional unit modules (such as the number of PEs and the number of multipliers included in a PE) are not fixed to a specific number and can be adaptively adjusted according to the implementation environment.

[0105] Figure 13 This is a diagram illustrating the filling process according to this disclosure.

[0106] Reference Figure 13 Methods for sending input in image form can include block-based and line-based methods. In one implementation, real-time AI acceleration hardware can be implemented by applying a line-based input method, which is relatively advantageous for input padding compared to block-based input methods.

[0107] For example, in Figure 13 In the diagram, the left figure can correspond to input data of size 12×12. Furthermore, the right figure can represent the process of padding after the input is sent in blocks or rows. In this case, the individual numbers in the figures can represent the input data divided into blocks or rows.

[0108] More specifically, in the upper right image, the pixels on the top and left sides of input block 1 can be included in the edge region of the original input data. Therefore, padding may be required for the top and left sides of input block 1. Additionally, for input blocks 2, 3, and 4, padding may be required for the top, top and right sides, and left side, respectively. In the case of input block 5, since pixels are not included in the edge region of the original input data, padding is not necessary. Furthermore, for input blocks 6, 7, and 8, padding may be required for the right side, bottom side, and bottom and right side, respectively. As a result, depending on the overlap between the edge regions of the input blocks and the original input data, a total of nine padding methods may be required in the block-based input method.

[0109] Furthermore, in the lower right figure, when an input line exists at a padding position such as input line 1, padding can be performed on the corresponding input line. Additionally, in the case of input line 2, padding can be performed only on the beginning of the input line, while in the case of input line 3, padding can be performed only on the end. In the case of input line 4, padding may not be necessary. Therefore, for the line-based input method, a total of four padding methods may be required. Since fewer methods are needed compared to the nine padding methods required in the block-based input method, the line-based input method is superior to the block-based input method in terms of padding simplicity.

[0110] Figure 14 This is a diagram illustrating the filling process for line-based input according to this disclosure.

[0111] Reference Figure 14 Real-time AI-accelerated hardware can perform padding processing using a row-based input method. More specifically, the real-time AI-accelerated hardware can detect whether the first row of the input feature map has been read before reading the row buffer.

[0112] If yes, the real-time AI acceleration hardware can begin padding the entire line buffer. If no, the real-time AI acceleration hardware can detect whether the line buffer includes edge regions of the input feature map. If it includes edge regions, the real-time AI acceleration hardware can detect whether the edge region corresponds to the top or bottom and can perform padding on the corresponding portion. If it does not include edge regions, the real-time AI acceleration hardware may not perform padding on the corresponding portion.

[0113] Figure 15 This is a diagram illustrating an embodiment of the real-time artificial intelligence acceleration hardware device according to the present disclosure.

[0114] Reference Figure 15 The real-time AI acceleration hardware device 1530 can be implemented to operate in conjunction with a host PC 1510. For example, Figure (a) can correspond to the case where the host PC 1510 and the real-time AI acceleration hardware device (1530) are implemented as independent devices but operate in conjunction with each other. In this case, the real-time AI acceleration hardware device 1530 can access the internal memory 1511 of the host PC 1510 to receive data for neural network operations and can store the results of neural network operations. That is, the real-time AI acceleration hardware device 1530 can access the internal memory 1511 corresponding to off-chip memory in conjunction with the host PC 1510, and can minimize access to off-chip memory by using on-chip memory 1531 during the neural network operation process.

[0115] Figure 15 This is a diagram illustrating the skip connection structure of a serial buffer according to the present disclosure.

[0116] Reference Figure 15 The real-time artificial intelligence acceleration hardware device 500 may include neural network units connected according to a neural network structure to implement the operation of the neural network. For example, the neural network unit may include an encoding unit for encoding processing during the operation of the neural network, a decoding unit for decoding processing, and a bottleneck unit connecting the encoding unit and the decoding unit.

[0117] In one embodiment, the real-time artificial intelligence acceleration hardware device 500 may include a concatenation buffer that directly connects the encoding and decoding units within a neural network. Here, the concatenation buffer can physically implement skip connections in the neural network structure. Specifically, the concatenation buffer can directly connect the encoding and decoding units according to the depth of the neural network, and can serve to send the concatenated feature map generated in the encoding unit to the decoding unit. For example, in... Figure 15 In this context, EN 0 unit can be connected to DE 0 unit via a serial buffer, and EN 1 unit can be connected to DE 1 unit via a serial buffer.

[0118] In one implementation, the cascaded buffers can be connected sequentially in a number independently calculated based on the depth of the neural network structure. For example, in Figure 15 In this configuration, the EN 0 unit can be connected to the DE 0 unit through three consecutive serial buffers, and the EN 1 unit can be connected to the DE 1 unit through one serial buffer.

[0119] In one implementation, the number of concatenated buffers can be calculated using the following formula, based on the number of downsampling operations performed in the encoding unit.

[0120] [Equation]

[0121] The number of CatBufs = 2(Dn) - 1

[0122] Here, D is the number of downsampling operations, i.e., the number of coding units, and n (≥0) is the index of each unit. For example, when in Figure 5 When performing two downsampling operations, a total of two coding units can be arranged. In the case of the first coding unit, n can be set to 0, and in the case of the second unit, n can be set to 1.

[0123] In this case, the number of connections in the concatenation buffer may need to be set appropriately based on the neural network architecture. That is, when the number of connections is relatively large, more concatenated feature maps than are needed can be stored, thus increasing the hardware size.

[0124] Conversely, when the number of connections is relatively small, the following problem may arise where the next concatenated feature map is stored before the current one is loaded. For example, in Figure 15 In this example, based on the neural network structure, the units can be arranged from EN 0 units to EN k units (where k is a natural number) according to the depth of the neural network. EN 0 units can be connected to DE 0 units via cascading buffers, and the number of cascading buffers can be calculated as 2(k+1-0)-1, since the total number of downsampled units in the entire neural network is D=k+1. The number of cascading buffers connected to EN 1 units can be calculated as 2(k+1-1)-1. As another example, in... Figure 5 In the case of En / De 0, the size of the serial buffer can be calculated as 2(2-0)-1=3, and the size of the serial buffer of En / De[1] can be calculated as 2(2-1)-1=1.

Claims

1. An artificial intelligence acceleration hardware device, the artificial intelligence acceleration hardware device comprising: Multiple neural network units, which are connected according to a neural network structure to realize the operation of the neural network; A controller configured to perform data exchange with off-chip memory and control the operation of the neural network units; as well as An on-chip memory includes: a plurality of row buffers configured to provide inputs to the neural network; a plurality of weight buffers configured to store weight values ​​used during operation of the neural network units; and an output buffer configured to store the outputs of the neural network.

2. The artificial intelligence acceleration hardware device according to claim 1, in, The neural network unit includes: a plurality of encoding units, which are configured to perform encoding processing in a neural network having an encoder-decoder structure to compress input data into encoded data; Multiple decoding units, configured to perform decoding processing to recover the encoded data; and A bottleneck unit, configured to connect the encoding unit and the decoding unit.

3. The artificial intelligence acceleration hardware device according to claim 2, wherein, Each neural network unit in the neural network unit includes: A shifter module configured to perform a shift operation on input data; A weight register module, configured to receive weights required for convolution operations from the weight buffer; A temporary buffer (TempBuf) module, configured to store intermediate feature maps generated during the convolution operation; and A convolutional unit (convolutional unit) module, which is configured to implement at least one convolutional layer that performs convolution operations.

4. The artificial intelligence acceleration hardware device according to claim 3, wherein, Each coding unit in the coding unit also includes a pool buffer module, which is configured to perform downsampling on the convolution results generated by the convolution unit module.

5. The artificial intelligence acceleration hardware device according to claim 3, wherein, The convolutional unit module includes: At least one processing element PE module, the at least one PE module including a plurality of multipliers; At least one adder, configured to sum the results of multiplication operations performed by the plurality of multipliers; and At least one Tsum module is configured to store the summation result generated by the at least one adder.

6. The artificial intelligence acceleration hardware device according to claim 5, in, The convolutional unit module is configured such that the number of multipliers, the number of at least one PE module, the number of at least one adder, and the number of at least one Tsum module can be variably applied based on at least one of the number of input channels, the number of output channels, and the filter size set in the convolutional layer.

7. The artificial intelligence acceleration hardware device according to claim 1, wherein, The controller is configured to access the off-chip memory to read input data and weight values ​​for the operation of the neural network before the operation of the neural network begins, and to access the off-chip memory to store output data generated by the operation of the neural network after the operation of the neural network is completed.

8. The artificial intelligence acceleration hardware device according to claim 1, wherein, The controller is configured to initiate a buffer data filling operation when the buffer data stored in the row buffer overlaps with the edge region of the input data.

9. The artificial intelligence acceleration hardware device according to claim 1, wherein, The on-chip memory also includes multiple serial buffers (CatBufs) configured to connect to the neural network units and store serial feature maps generated during the operation of the neural network units.

10. The artificial intelligence acceleration hardware device according to claim 9, wherein, Each row buffer in the row buffer is configured to store input data read from the off-chip memory on a row-by-row basis, and to provide the input data on a row-by-row basis to the neural network unit according to the control signal of the controller.

11. The artificial intelligence acceleration hardware device according to claim 9, wherein, Each weight buffer in the weight buffer is independently connected to the corresponding neural network unit and is configured to store all weight values ​​used during the operation of the corresponding neural network unit.

12. The artificial intelligence acceleration hardware device according to claim 9, wherein, The concatenation buffer directly connects the encoding and decoding units in the neural network unit, and connects them sequentially in a number independently calculated based on the depth of the neural network.

13. A method executed by an artificial intelligence acceleration hardware device, the artificial intelligence acceleration hardware device comprising a plurality of neural network units connected according to a neural network structure, the method comprising the following steps: Multiple weight values ​​used during the operation of the neural network unit are read from off-chip memory and stored in multiple weight buffers (WeightBuf) independently connected to the neural network unit; The neural network unit receives input from the off-chip memory through multiple row buffers (row buffers) on a row-by-row basis; The convolution operations for the operation of the neural network are executed sequentially by the neural network units; The result of the convolution operation is stored as the output of the neural network through an output buffer (output buffer); as well as The output buffer reads multiple outputs of the neural network and stores the multiple outputs in the off-chip memory.

14. The method according to claim 13, wherein, The steps of sequentially performing the convolution operation include: performing a padding operation on the buffer data provided by the row buffer when the buffer data, which is provided in rows, overlaps with the edge region of the input data.

15. A method executed by an artificial intelligence acceleration hardware device, the artificial intelligence acceleration hardware device comprising a plurality of neural network units connected according to a neural network structure, the method comprising the following steps: The neural network unit receives input from off-chip memory through multiple row buffers (row buffers) on a row-by-row basis; Convolution operations for encoding processing are performed through multiple encoding units within the neural network unit; Multiple concatenated feature maps generated by the encoding unit are stored using multiple concatenated buffers (CatBuf); The decoding unit in the neural network unit receives the concatenated feature map from the concatenation buffer and performs a convolution operation for decoding processing. The result of the decoding process is stored as multiple outputs of the neural network through an output buffer (output buffer); as well as The plurality of outputs of the neural network are read from the output buffer and stored in the off-chip memory.

Images