Memory devices for artificial neural networks

Abstract

A memory device for artificial neural networks (ANNs) includes at least one array of memory elements having N columns and M rows; and a memory controller configured to sequentially perform read or write operations on data of the at least one array of memory elements in a burst mode based on predetermined sequential access information. Each of the at least one array of memory elements may include a plurality of dynamic memory elements with leakage current characteristics. The memory device may also include a processor configured to provide the memory controller with ANN data locality information or information for identifying input feature maps, kernels, and output feature maps. The memory controller can prepare the data of the ANN model for processing at the processor-memory level before being requested by the processor, thereby significantly reducing the latency of memory data provided to the processor.

Description

Technical Field

[0001] This disclosure relates to a memory device, and more specifically, to a memory device for artificial neural networks. Background Technology

[0002] With the development of artificial intelligence reasoning capabilities, various reasoning services, such as voice recognition, speech recognition, image recognition, object detection, driver drowsiness detection, dangerous moment detection, and gesture detection, are being integrated into various electronic devices. Electronic devices with reasoning services can include, for example, AI speakers, smartphones, smart refrigerators, VR devices, AR devices, AI closed-circuit television, as well as AI robotic cleaners, tablets, laptops, autonomous vehicles, bipedal robots, quadrupedal robots, and industrial robots.

[0003] Recently, with the development of deep learning technology, the performance of artificial neural network inference services based on big data learning has improved. Artificial neural network learning and inference services repeatedly train artificial neural networks with massive amounts of learning data and use the trained artificial neural network models to perform inference on various complex data. Therefore, various services can be provided to the aforementioned electronic devices by utilizing artificial neural network technology.

[0004] However, the functionality and accuracy required for inference services utilizing artificial neural networks are steadily increasing. Correspondingly, the size of artificial neural network models, computational demands, and the size of training data are growing exponentially. The performance requirements for processors and memory capable of handling inference operations of artificial neural network models are also gradually increasing. Furthermore, artificial neural network inference services are being proactively offered to cloud-based servers that can easily handle large amounts of data.

[0005] Simultaneously, research is actively underway on edge computing utilizing artificial neural network models. Edge computing refers to the edge or periphery where computation takes place. Therefore, edge computing refers to terminals that directly generate data or various electronic devices adjacent to these terminals, and can also be called edge devices. Edge devices can be used to perform necessary tasks immediately and reliably, such as those autonomous drones, robots, or vehicles that need to process large amounts of data within 1 / 100th of a second. Therefore, edge devices are suitable for fields where their numbers are rapidly increasing. Summary of the Invention

[0006] The inventors of this disclosure have recognized that traditional artificial neural network models suffer from problems such as high power consumption, heat generation, and processor bottlenecks caused by low memory bandwidth and memory latency. Therefore, the inventors also recognize the various difficulties in improving the operational processing performance of artificial neural network models and the need to develop an artificial neural network memory system capable of mitigating these problems.

[0007] Therefore, the inventors of this disclosure have investigated an artificial neural network (ANN) memory system suitable for server systems and / or edge computing. Furthermore, the inventors of this disclosure have also investigated a neural processing unit (NPU), which is a processor for an ANN memory system optimized for processing artificial neural network (ANN) models.

[0008] First, the inventors of this disclosure recognize that, in order to improve the computational processing speed of artificial neural networks, it is crucial to effectively control the memory during the computation process of the artificial neural network model. The inventors of this disclosure have recognized that when an artificial neural network model is being trained or inferred, if the memory is not properly controlled, the necessary data is not prepared in advance, resulting in a reduction in the effective bandwidth of the memory and / or a delay in the supply of data to the memory. Furthermore, the inventors of this disclosure have recognized that in this situation, a starvation or idle state can occur due to the lack of data to be processed by the processor, making it impossible to perform actual operations and thus leading to a decrease in operational performance.

[0009] Secondly, the inventors of this disclosure have recognized the limitations of the operational processing methods for artificial neural network models at the algorithmic level of known technologies. For example, a known prefetching algorithm is a technique that analyzes an artificial neural network model at the conceptual layer unit level so that the processor reads data from memory at each layer unit. However, prefetching algorithms cannot identify the locality of artificial neural network data in word units or memory access request units of the artificial neural network model at the processor-memory level (i.e., the hardware level). The inventors of this disclosure have recognized that it is difficult to optimize data transmission / reception operations at the processor-memory level using prefetching techniques alone.

[0010] Third, the inventors of this disclosure have recognized "artificial neural network data locality," a unique characteristic of artificial neural network models. The inventors of this disclosure have recognized that artificial neural network data locality exists in processor-memory level word units or memory access request units, and that effective memory bandwidth is maximized and the latency of data provided to the processor is minimized by utilizing artificial neural network data locality, thereby improving the processor's artificial neural network learning / inference operation processing performance.

[0011] Specifically, the "artificial neural network data locality" as understood by the inventors of this disclosure refers to the sequence information of word units of data required by the processor to perform computational processing on the artificial neural network. This sequence is executed according to the structure and operational algorithms of the artificial neural network model when the processor processes a specific artificial neural network model. Furthermore, the inventors of this disclosure have recognized that artificial neural network data locality is maintained within the operational processing sequence of the artificial neural network model for iterative learning and / or inference operations provided to the processor. Accordingly, the inventors of this disclosure have recognized that while maintaining artificial neural network data locality, the processing sequence of data required by the processor for the artificial neural network operations is maintained within word units, and information to be used for artificial neural network operations is provided or analyzed. In other words, a word unit of the processor can refer to a component unit that serves as the basic unit to be processed by the processor. For example, when a neural processing unit processes N-bit input data multiplied by M-bit kernel weights, the processor's input data word unit can be N bits, and the weight data word unit can be M bits. Furthermore, the inventors of this disclosure have recognized that the processor's word units can be set differently depending on the layers, feature maps, kernels, activation functions, etc., of the artificial neural network model. Accordingly, the inventors of this disclosure also recognize that precise memory control technology is necessary for operations performed at word units.

[0012] The inventors of this disclosure have noted that artificial neural network data locality is constructed when an artificial neural network model is compiled by a compiler for execution on a specific processor. Furthermore, the inventors have recognized that artificial neural network data locality can be constructed based on the operational characteristics of the algorithms applied to the compiler, the artificial neural network model, and the processor architecture. Additionally, the inventors of this disclosure have recognized that even within the same artificial neural network model, the artificial neural network data locality of the model to be processed can be constructed in various forms, depending on the computational method of the artificial neural network model by the processor, such as feature map tiling, the stabilization technique of the processing elements, the number of processing elements in the processor, the feature maps in the processor, the capacity of cache memory such as weights, the memory hierarchy in the processor, or the algorithmic features of the compiler that determine the sequence of computational operations of the processor to compute the artificial neural network model. This is because even when computing the same artificial neural network model, the processor will determine that the data sequence required at each moment in the clock unit is different due to the factors mentioned above. That is, the inventors of this disclosure have recognized that the sequence of data necessary for the computation of an artificial neural network model is conceptually a sequence of computations of layers, unit convolutions, and / or matrix multiplications of the artificial neural network. Furthermore, the inventors of this disclosure have recognized that, in the data sequences required for physical computation, the artificial neural network data locality of an artificial neural network model is constructed at the processor-memory level (i.e., the hardware level) in word units. In addition, the inventors of this disclosure have recognized that artificial neural network data locality depends on the processor and the compiler used by the processor.

[0013] Fourth, the inventors of this disclosure have recognized that when an artificial neural network memory system is provided that is configured to provide information on the locality of artificial neural network data in order to utilize the locality of artificial neural network data, the processing performance of the artificial neural network model can be maximized at the processor-memory level.

[0014] The inventors of this disclosure have recognized that when an artificial neural network memory system accurately computes word units of the artificial neural network model's data locality, the processor also discovers the operation processing sequence information of that word unit, which is the smallest unit by which the processor processes the artificial neural network model. In other words, the inventors of this disclosure have recognized that when an artificial neural network memory system utilizing the data locality of artificial neural networks is provided, the artificial neural network memory system can accurately predict whether specific data will be read from memory at a specific time to provide that specific data to the processor, or whether that specific data needs to be computed by the processor to store that specific data in memory at a specific time in word units. Therefore, the inventors of this disclosure have recognized that providing an artificial neural network system allows for the pre-preparation of data to be requested by the processor in word units.

[0015] In other words, the inventors of this disclosure have recognized that if the artificial neural network memory system is aware of the data locality of the artificial neural network, then when the processor uses techniques such as feature map tiling to compute a convolution of a specific input data and a specific kernel, the sequence of operations for processing the convolution as the kernel moves in a specific direction is also known in the word unit.

[0016] In other words, it is recognized that artificial neural network memory systems predict which data is necessary for the processor by utilizing the data locality of artificial neural networks, thereby predicting memory read / write operations to be requested by the processor, and that the data to be processed by the processor is prepared in advance to minimize or eliminate increases in effective memory bandwidth and / or memory data supply latency. Furthermore, the inventors have recognized that processor starvation or idle states can be minimized when the artificial neural network memory system provides data to be processed by the processor in the necessary timing sequence. Therefore, the inventors of this disclosure have recognized that operational processing performance can be improved and power consumption can be reduced through artificial neural network memory systems.

[0017] Fifth, the inventors of this disclosure have recognized that even if artificial neural network (ANN) data locality information may not be provided to the ANN memory controller, after the ANN memory controller is placed in the communication channel between the processor and memory that processes the ANN model, when the processor processes the operation of a specific ANN model, the data access requests to the memory are analyzed to infer the ANN data locality of the ANN model being processed by the processor in the data access request unit between the processor and memory. That is, the inventors of this disclosure have recognized that each ANN model has a unique ANN data locality, thereby the processor generates a specific sequence of data access requests at the processor-memory level based on the ANN data locality. Furthermore, the inventors of this disclosure have recognized that the access queue of data stored in memory for data requests between the processor and memory is based on maintaining ANN data locality while the processor iteratively processes the learning / inference operations of the ANN model.

[0018] Therefore, the inventors of this disclosure have placed an artificial neural network memory controller in the communication channel of a processor that operates the artificial neural network model and the memory. Furthermore, by observing data access requests between the processor and the memory for one or more learning and inference operations, the inventors have recognized that the artificial neural network memory controller can infer the data locality of the artificial neural network within the data access request unit. Therefore, the inventors of this disclosure have recognized that even without providing information on the data locality of the artificial neural network, the data locality of the artificial neural network can be inferred by the artificial neural network memory controller.

[0019] Therefore, the inventors of this disclosure have recognized that, based on the data locality of artificial neural networks reconstructed in the data access request unit, memory read / write operations requested by the processor can be predicted, and the increase in effective memory bandwidth and / or the delay in memory data supply can be minimized or substantially eliminated by pre-preparing the data to be processed by the processor. Furthermore, the inventors of this disclosure have recognized that the occurrence rate of processor starvation or idle states can be minimized when the artificial neural network memory system provides the data to be processed by the processor in the necessary timing.

[0020] Therefore, the objective of this disclosure is to provide an artificial neural network (ANN) memory system that optimizes the artificial neural network operation of a processor by utilizing the data locality of an artificial neural network (ANN) model operating at the processor-memory level.

[0021] Therefore, the problem to be solved by the present invention is to provide an artificial neural network memory system, which includes an artificial neural network memory controller capable of reducing memory latency by (1) analyzing multiple data access requests generated by the processor and (2) pre-preparing data access requests to be requested by the processor from the data locality patterns of the artificial neural network model being processed by the processor. However, this disclosure is not limited thereto, and other problems will be clearly understood by those skilled in the art from the following description.

[0022] According to one aspect of this disclosure, a memory device for artificial neural networks (ANNs) is provided. The memory device may include: at least one array of memory cells having N columns and M rows; and a memory controller configured to sequentially perform read or write operations on data of the at least one array of memory cells in a burst mode based on predetermined sequential access information.

[0023] Each of the at least one memory element array may include multiple dynamic memory elements with leakage current characteristics.

[0024] Each of the at least one memory element array may include: a column decoder for controlling access to the N columns; multiple bit lines connected to the column decoder; a row decoder for controlling access to the M rows; multiple word lines connected to the row decoder; and a sense amplifier connected to one end of each of the multiple bit lines.

[0025] The at least one memory element array can store data required for the operation of the artificial neural network. The memory controller is also configured to control data communication between the processor and the at least one memory element array. The processor is configured to process the artificial neural network operation based on the predetermined sequential access information.

[0026] The predetermined sequential access information can be generated based on the ANN data locality information of the artificial neural network.

[0027] The memory controller can also be configured to directly control the addresses of the N columns and M rows of the at least one memory element array, such that the at least one memory element array operates sequentially in burst mode based on the predetermined sequential access information.

[0028] The memory controller can also be configured to set the memory address of data for each operational step to be stored in the at least one memory element array based on the sequential access information.

[0029] The memory controller can also be configured to store the data of the artificial neural network by sequentially assigning addresses corresponding to the N columns and M rows of the at least one memory element array.

[0030] Furthermore, according to another aspect of this disclosure, a memory device for an artificial neural network (ANN) is provided. The memory device may include: at least one array of memory elements; and a memory controller configured to directly control read or write operations of the at least one array of memory elements based on ANN data locality information of the artificial neural network.

[0031] The ANN data locality information may include the predetermined operation sequence information of the artificial neural network.

[0032] The ANN data locality information may include the data size information of each operation in a preset operation sequence.

[0033] The memory controller can also be configured to store a memory map. The memory map is configured sequentially based on operation sequence information and the data size of each operation sequence.

[0034] The ANN data locality information may include signals used to identify weights, input feature maps, and output feature maps. The pattern of the operation sequences of the weights, input feature maps, and output feature maps may be determined through compilation based on the characteristics of the processor.

[0035] The locality information of the ANN data can be determined based on at least one of the characteristics of the artificial neural network model, the characteristics of the processor, the size of the cache memory, and the operation algorithm strategy.

[0036] Furthermore, according to another aspect of this disclosure, a memory device for an artificial neural network is provided. The memory device may include: at least one array of dynamic memory elements; and a memory controller configured to store data of the artificial neural network in the at least one array of dynamic memory elements according to a sequence based on ANN data locality information.

[0037] The sequence based on the ANN data locality information may include a repeating pattern having an order of input feature map, kernel, and output feature map.

[0038] The sequence based on the ANN data locality information may include a repeating pattern with a kernel, input feature map, and output feature map order.

[0039] The ANN data locality information can be configured within a unit that receives a data access request from the processor and is sent to the memory controller. The ANN data locality information may include sequence information about all data access requests required to perform the inference operations of the artificial neural network.

[0040] The memory controller can also be configured to divide each of the at least one dynamic memory element array into a kernel region and a feature map region based on information used to identify the kernel, input feature map, and output feature map.

[0041] Each of the at least one dynamic memory element array may include a plurality of memory banks configured to enable interleaving operations. The memory controller is also configured to partition and store the data of the artificial neural network in each of the plurality of memory banks to operate in burst mode corresponding to the interleaving operations of the plurality of memory banks.

[0042] The memory device may also include a processor configured to provide the ANN data locality information to the memory controller.

[0043] The memory device may also include a processor configured to provide the memory controller with information for identifying the input feature map, the kernel, and the output feature map.

[0044] According to examples of this disclosure, in systems that process artificial neural networks, the latency of providing data from memory to the processor can be substantially eliminated or reduced through the data locality of artificial neural networks.

[0045] According to examples in this disclosure, an artificial neural network memory controller can prepare data for an artificial neural network model to be processed at the processor-memory level before being requested by the processor.

[0046] According to examples of this disclosure, the processing time for learning and inference operations of artificial neural network models processed by a processor is reduced, thereby improving the processor's operational processing performance and increasing the power efficiency of operational processing at the system level.

[0047] The effects of this disclosure are not limited to those illustrated above, and this specification includes many more effects. Attached Figure Description

[0048] Figure 1A This is a schematic block diagram of an artificial neural network memory system according to an example of this disclosure.

[0049] Figure 1B This is a schematic diagram illustrating an exemplary neural processing unit for explaining the reconstruction of artificial neural network data locality patterns applicable to various examples of this disclosure.

[0050] Figure 2 This is an illustration based on examples from this disclosure used to explain patterns of data locality in artificial neural networks.

[0051] Figure 3 This is a schematic diagram illustrating an exemplary artificial neural network model used to explain patterns of data locality in artificial neural networks that can be applied to various examples of this disclosure.

[0052] Figure 4 Examples based on this disclosure are used to explain the analysis via an artificial neural network memory controller. Figure 3 A schematic diagram of the data locality pattern of artificial neural networks generated by artificial neural network models.

[0053] Figure 5 It is an explanation and Figure 4 A diagram illustrating the tokens and recognition information corresponding to the locality patterns of artificial neural network data.

[0054] Figure 6 This is an example of an example provided in this disclosure used to explain the predicted data access requests and subsequent data access requests generated by an artificial neural network memory controller based on the data locality pattern of an artificial neural network.

[0055] Figure 7 This is a flowchart illustrating the operation of an artificial neural network memory controller according to an example of this disclosure.

[0056] Figure 8 This is a schematic block diagram of an artificial neural network memory system according to another example of this disclosure.

[0057] Figure 9 This is a schematic diagram illustrating the operation of a memory system according to a comparative embodiment of the present disclosure.

[0058] Figure 10 yes Figure 8 A schematic diagram of the operation of the memory system.

[0059] Figure 11 This is a schematic block diagram of an artificial neural network memory system according to yet another example of this disclosure.

[0060] Figure 12 This is an illustration of exemplary identification information for a data access request.

[0061] Figure 13 It is a diagram used to explain the energy consumption of each unit operation in an artificial neural network memory system.

[0062] Figure 14 These are schematic diagrams illustrating various examples of artificial neural network memory systems based on this disclosure.

[0063] Figure 15 This is an example diagram showing a substrate and a channel on which the memory is mounted.

[0064] Figure 16 This is an example diagram illustrating the process of reading data from a memory with a multi-memory structure.

[0065] Figure 17 This is an example diagram illustrating the delay that occurs in conventional DRAM.

[0066] Figure 18 This is an example diagram illustrating the basic concept of Sequential Access Memory (SAM) according to this disclosure.

[0067] Figure 19 This is an example table showing the computational load and data size for 16 layers.

[0068] Figure 20 This is an example table showing the computational load and data size for 28 layers.

[0069] Figure 21 This is a table illustrating a first example of accessing memory based on sequence information in Artificial Neural Network Data Locality (ANN DL) information.

[0070] Figure 22 It is shown in a simplified way Figure 21 Example table of tables.

[0071] Figure 23 SAM was shown according to Figure 22An example of setting up a memory address mapping in a table.

[0072] Figure 24 This is a table illustrating a second example of accessing memory based on sequence information in ANN DL information.

[0073] Figure 25 SAM was shown according to Figure 24 An example of setting up a memory address mapping in a table.

[0074] Figure 26 This is a table illustrating a third example of accessing memory based on sequence information in ANN DL information.

[0075] Figure 27A and 27B An example of setting up memory address mapping based on ANN DL information is shown.

[0076] Figure 28 This is a conceptual diagram illustrating the control signals of the SAM controller.

[0077] Figure 29 The diagram is based on Figure 28 Example diagram of how sideband signals set memory address mapping.

[0078] Figure 30A An example of setting memory address mapping based on sideband signals is shown. Figure 30B An example is shown in which only the kernel's memory address mapping is set sequentially.

[0079] Figure 31A This is an example diagram illustrating the READ_DISCARD command transmitted via sideband signals according to an example of this disclosure, and Figure 31B An example of the READ command is shown.

[0080] Figure 32 A portion of a circuit diagram of an exemplary SAM implemented as a memory element of DRAM according to an example of this disclosure is shown.

[0081] Figure 33 It is used for explanation Figure 32 An example diagram of the pre-charge operation in a SAM circuit diagram.

[0082] Figure 34 It is used for explanation Figure 32 An example diagram of memory element access operations in a SAM circuit diagram.

[0083] Figure 35 It is used for explanation Figure 32 An example diagram of data sensing operation in a SAM circuit diagram.

[0084] Figure 36 It is used for explanation Figure 32 An example diagram of the READ-DISCARD operation in a SAM circuit diagram.

[0085] Figure 37 It is used for explanation Figure 32 Example diagram of the READ operation in the SAM circuit diagram.

[0086] Figure 38A This is an example waveform diagram of the READ-DISCARD operation. Figure 38B This is an example waveform diagram of the READ operation.

[0087] Figure 39 It is shown Figure 21 A portion of the table used to explain REFREASH operations.

[0088] Figure 40 Examples of SAM memories implemented in various forms according to the present disclosure are shown.

[0089] Figure 41 This is an example diagram illustrating a method for mapping main memory addresses based on ANN DL information.

[0090] Figure 42 This is an example diagram illustrating another example of a method for mapping main memory addresses based on ANN DL information.

[0091] Figure 43 This is a table illustrating an example of accessing memory based on sequence information in the ANN DL information.

[0092] Figure 44 This is an example diagram illustrating a memory in which a SAM controller is embedded.

[0093] Figure 45 This is an example diagram illustrating the architecture of the compiler.

[0094] Figure 46 The architecture based on the first example is shown.

[0095] Figure 47 The architecture based on the second example is shown.

[0096] Figure 48 The architecture based on the third example is shown.

[0097] Figure 49 The architecture based on the fourth example is shown.

[0098] Figure 50 The architecture based on the fifth example is shown.

[0099] Figure 51The architecture based on the sixth example is shown.

[0100] Figure 52 The diagram is based on Figure 51 The sixth example shown is an example diagram of the operation.

[0101] Figure 53A and 53B This is an example diagram illustrating an example of convolution.

[0102] Figure 54 Another example is shown where operations are performed based on tiling techniques after data is cached from main memory to cache memory.

[0103] Figure 55 This is a schematic diagram illustrating an artificial neural network memory system according to various examples of the present disclosure.

[0104] Figure 56 It shows Figure 55 Detailed configuration of the SFU.

[0105] Figure 57 A graph showing the bandwidth of the data bus between the buffer memory (cache) and the main memory is shown. Detailed Implementation

[0106] The advantages and features of this disclosure, as well as methods for achieving these advantages and features, will become clear from the various examples described in detail below and the accompanying drawings. However, the invention is not limited to the examples disclosed herein, but will be implemented in various forms. Examples are provided so that the invention can be fully disclosed and its scope readily understood by those skilled in the art. Therefore, the invention will be defined only by the scope of the appended claims.

[0107] For ease of description, specific examples are used to illustrate the detailed description of this disclosure with reference to the accompanying drawings, which serve as examples of how this disclosure can be implemented. Although the components of the various examples of this disclosure differ from one another, the manufacturing methods, operating methods, algorithms, shapes, processes, structures, and features described in a particular example may be combined with or included in other embodiments. Furthermore, it should be understood that the position or arrangement of the various components in each disclosed example may be changed without departing from the spirit and scope of this disclosure. The features of the various embodiments of this disclosure may be combined or integrated with each other in part or in whole, and may be interlocked and operated in various technically sophisticated ways that will be understood by those skilled in the art, and the embodiments may be performed independently of each other or in relation to each other.

[0108] The shapes, dimensions, ratios, angles, quantities, etc., shown in the accompanying drawings to describe the examples of this disclosure are merely examples, and this disclosure is not limited thereto. Throughout the specification, the same reference numerals denote the same elements. Furthermore, in the following description, detailed explanations of known related technologies are omitted to avoid unnecessarily obscuring the subject matter of this disclosure. Terms such as “comprising,” “having,” and “consisting of” as used herein are generally intended to allow for the addition of additional components, unless these terms are used in conjunction with the term “only.” Unless explicitly stated otherwise, any singular reference may include the plural. Even without explicit statement, a component is construed as including a general range of tolerances. When the positional relationship between two elements is described using terms such as “on,” “above,” “below,” “near,” or “adjacent,” a component may be located between the two components, unless these terms are used in conjunction with the terms “immediately following” or “directly.” When an element or layer is disposed “on” another element or layer, the other layer or element may be directly inserted onto or between the other element.

[0109] Figure 1A The example illustration of this disclosure is an artificial neural network memory system 100 based on the data locality of an artificial neural network.

[0110] refer to Figure 1A The artificial neural network memory system 100 can be configured to include at least one processor 110 and at least one artificial neural network memory controller 120. That is, at least one processor 110 according to the example of this disclosure is provided, and multiple processors can be used. At least one artificial neural network memory controller 120 according to the example of this disclosure is also provided, and multiple artificial neural network memory controllers can be used.

[0111] For ease of description, in the following text, when at least one processor 110 includes only one processor, it may be referred to as processor 110.

[0112] For ease of description, in the following text, when at least one artificial neural network memory controller 120 includes only one artificial neural network memory controller 120, it can be referred to as artificial neural network memory controller 120.

[0113] Processor 110 is configured to process artificial neural network models. For example, processor 110 processes inference of an artificial neural network model trained to perform specific inference functions to provide inference results based on input data. For example, processor 110 processes the learning of an artificial neural network model used to perform specific inference functions to provide a trained artificial neural network model. Specific inference functions may include various inference functions that can be inferred by the artificial neural network, such as object recognition, speech recognition, and image processing.

[0114] Processor 110 may be configured to include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a digital signal processing device (DSP), an arithmetic and logic unit (ALU), and an artificial neural processing unit (NPU). However, processor 110 of this disclosure is not limited to the processors described above.

[0115] Processor 110 can be configured to communicate with artificial neural network memory controller 120. Processor 110 can be configured to generate data access requests. Data access requests can be sent to artificial neural network memory controller 120. Here, a data access request may refer to a request to access data required by processor 110 for processing inference or learning of an artificial neural network model.

[0116] The processor 110 can send a data access request to the artificial neural network memory controller 120 to be provided with the data required for inference or learning of the artificial neural network model from the artificial neural network memory controller 120, or to be provided to the artificial neural network memory controller 120 with the inference or learning results of the artificial neural network processed by the processor 110.

[0117] The processor 110 can provide inference or learning results obtained by processing a specific artificial neural network model. In this case, the processor 110 can be configured to process operations of the artificial neural network for inference or learning in a specific sequence.

[0118] The processor 110 needs to process the operations of the artificial neural network in a specific sequence because each artificial neural network model is configured to have a unique artificial neural network structure. That is, each artificial neural network model is configured to have unique artificial neural network data locality based on its unique artificial neural network structure. Furthermore, the sequence of operations of the artificial neural network model processed by the processor 110 is determined based on the unique artificial neural network data locality.

[0119] In other words, the data locality of an artificial neural network can be configured when the artificial neural network model is compiled by a compiler to execute on a specific processor. The data locality of an artificial neural network can be configured based on the algorithms applied to the compiler and the artificial neural network model, as well as the operating characteristics of the processor.

[0120] The artificial neural network model to be processed by processor 110 can be compiled by processor 110 and a compiler that takes into account the algorithmic characteristics of the artificial neural network model. That is, when the driving characteristics of processor 110 are known with knowledge of the structure and algorithmic information of the artificial neural network model, the compiler can be configured to provide artificial neural network data locality information to artificial neural network memory controller 120 in word unit order.

[0121] For example, the weight values ​​of a specific layer in a specific artificial neural network model at the algorithm level of known techniques can be calculated at the layer unit level. However, the weight values ​​of a specific layer in a specific artificial neural network model at the processor-memory level according to the examples of this disclosure can be calculated at the word unit level, which is intended to be processed by processor 110.

[0122] For example, when the size of the cache memory of processor 110 is smaller than the data size of the weights of a specific layer of the artificial neural network model to be processed, processor 110 can be compiled to not process all the weight values ​​of a specific layer at once.

[0123] In other words, when processor 110 calculates the weights and node values ​​of a specific layer, the cache memory storing the results may become insufficient due to the large size of the weights. In this case, the data access requests generated by processor 110 can increase into multiple data access requests. Therefore, processor 110 can be configured to process the increased data access requests in a specific order. In this case, the order of operations at the algorithm level and the order of operations based on the data locality of artificial neural networks at the processor-memory level can differ from each other.

[0124] In other words, by taking into account the hardware characteristics of the processor and memory, the algorithm-level artificial neural network operation sequence can be reconstructed through the processor-memory level artificial neural network data locality to process the corresponding artificial neural network model.

[0125] The data locality of an artificial neural network model existing at the processor-memory level can be defined as information that predicts the order of operations of the artificial neural network model to be processed by the processor 110 at the processor-memory level, based on the order of data access requests from the processor 110 to the memory.

[0126] In other words, even within the same artificial neural network model, the artificial neural network data locality can be configured differently depending on the operational functions of the processor 110, such as feature map tiling techniques or stationary techniques of processing elements, cache memory capacity, the number of processing elements in the processor 110, feature maps and weights in the processor 110, memory hierarchy in the processor 110, and the algorithmic characteristics of the compiler that determines the sequence of computational operations of the processor 110 to compute the artificial neural network model.

[0127] For example, feature map tiling is an artificial neural network technique that divides convolutions, and as the convolutional regions are divided, the feature maps are partitioned for computation. Therefore, due to tiling convolutions, even the same artificial neural network model can exhibit different artificial neural network data localities.

[0128] For example, stabilization techniques are techniques for controlling the driving methods of processing elements (PEs) in a neural processing unit. According to stabilization techniques, the data type to be processed, such as one of the input feature map, weights, and output feature map, is fixed to the processing element to be reused. Therefore, the type of data or sequence requested by the processor 110 from the memory can vary.

[0129] In other words, even within the same artificial neural network model, data locality can be reconstructed according to various algorithms and / or techniques. Therefore, data locality in artificial neural networks can be reconstructed, wholly or partially, under various conditions, such as processors, compilers, or memory.

[0130] Figure 1B An example of an exemplary neural processing unit is illustrated for explaining the reconstruction of artificial neural network data locality patterns applicable to various examples of this disclosure.

[0131] refer to Figure 1B The illustration shows an exemplary smoothing technique applicable when the processor 110 is a neural network processing unit (NPU).

[0132] An NPU can include multiple processing elements. Processing elements (PEs) can be configured as an array, and each processing element can be configured to include a multiplier (×) and an adder (+). Processing elements (PEs) can be connected to buffer memory or cache memory, such as a global buffer. A processing element (PE) can pin one of the following data to its registers: the input feature map pixel (Ifmap pixel: I), the filter weight W, and the partial sum (Psum: P). The remaining data can be provided as input data to the processing element (PE). When the accumulation of the partial sum P is complete, it may become the output feature map pixel. However, multiple processing elements can be implemented as individual drivers instead of as an array.

[0133] Figure 1B View (a) in the figure illustrates the Weight Smoothing (WS) technique. According to the WS technique, filter weights W0 through W7 are fixed to the respective register files of the processing element PE, and the input feature map pixels I, input in parallel to the processing element PE, are moved from the zeroth input feature map pixel I0 to the eighth input feature map pixel I8 to perform the operation. Parts P0 through P8 can be accumulated in the cascaded processing elements PE. Parts P0 through P8 can be sequentially moved to subsequent processing elements. All multiplication and accumulation (MAC) operations using fixed filter weights W0 through W7 need to be mapped to the same processing element PE for serial processing.

[0134] According to the above configuration, during the convolution operation of filter weights W in the register file, the reuse of filter weights W is maximized to minimize the energy consumption of accessing filter weights W.

[0135] It is important to note that since the WS technique is applied to the artificial neural network model during the compilation step, the artificial neural network data locality of the model is reconstructed to optimize for the WS technique at the processor-memory level. For example, according to the WS technique, for operational efficiency, filter weights W0 to W7 can be preferentially stored in the processing element PE. Accordingly, the artificial neural network data locality can be reconstructed in the order of filter weights W, input feature map pixels I, and partial sums P, thereby determining the data access request sequence generated by processor 110 based on the reconstructed artificial neural network data locality.

[0136] Figure 1BView (b) illustrates the Output Smoothing (OS) technique. According to the OS technique, segments P0 through P7 are fixed to the respective register files of the processing element PE to be accumulated, and the filter weights W input to the processing element PE in parallel are moved from the zeroth input filter weight W0 to the seventh filter weight W7 to perform the operation. Input feature map pixels I0 through I7 can be moved to the cascaded processing elements PE. Each segment and P0 through P7 needs to be fixed to each processing element PE to be mapped to perform the Multiplication and Accumulation (MAC) operation.

[0137] According to the above configuration, during the convolution operation of the filter weights W in the processing element PE, the partial sums P are fixed to the register file of the processing element PE to maximize the reuse of the partial sums P and minimize energy consumption based on the movement of the partial sums P. When the accumulation of the fixed partial sums P is completed, it can become the output feature map.

[0138] It should be noted that when the processor 110 applies the Output Smooth OS technique, the artificial neural network data locality of the artificial neural network model is reconstructed to optimize for the Output Smooth OS technique at the processor-memory level. For example, according to the Output Smooth OS technique, for operational efficiency, portions P0 to P7 are preferentially stored in the processing element PE. Accordingly, the artificial neural network data locality can be reconstructed in the order of portion P, filter weights W, and input feature map pixels I, thereby determining the data access request sequence generated by the processor 110 based on the reconstructed artificial neural network data locality. The artificial neural network model compiler receives hardware characteristic information from the processor 110 and memory to convert it into code for the artificial neural network model to operate at the processor-memory level. At this time, the artificial neural network model is converted into code executed by the processor, thereby converting the artificial neural network model into low-level code.

[0139] In other words, based on the factors mentioned above, even when processing the same artificial neural network model, the processor 110 can change the order of the data required at each moment in the clock unit. Therefore, the artificial neural network data locality of the artificial neural network model can be configured to be different at the hardware level.

[0140] However, once the configuration of data locality in the artificial neural network is complete, the operating sequence of the processor 110 and the data processing sequence required for the operation can be accurately repeated during each learning or inference operation of the corresponding artificial neural network model.

[0141] In the following, the artificial neural network memory system 100 described above according to the example of this disclosure can be configured to predict the next data to be requested by the processor 110 based on the accurate operating sequence provided by the data locality of the artificial neural network, so as to improve memory latency and memory bandwidth problems, thereby improving the operational processing performance of the artificial neural network and reducing power consumption.

[0142] According to an example of this disclosure, the artificial neural network memory controller 120 is configured to be provided with artificial neural network data locality information of an artificial neural network model to be processed by the processor 110, or to be configured to analyze the artificial neural network data locality of an artificial neural network model being processed by the processor 110.

[0143] The artificial neural network memory controller 120 can be configured to receive data access requests generated by the processor 110.

[0144] The artificial neural network memory controller 120 can be configured to monitor or record data access requests received from the processor 110. The artificial neural network memory controller 120 observes data access requests output by the processor 110, which is processing an artificial neural network to accurately predict the data access queue that will be requested later. A data access request can be configured to include at least one word unit of data.

[0145] The artificial neural network memory controller 120 can be configured to sequentially record or monitor data access requests received from the processor 110.

[0146] The data access requests recorded by the artificial neural network memory controller 120 may be stored in various forms, such as log files, tables, or lists. However, the artificial neural network memory controller 120 according to the examples of this disclosure is not limited to the type or form of data access requests recorded.

[0147] Data access requests monitored by the artificial neural network memory controller 120 can be stored in any memory within the artificial neural network memory controller 120. However, the artificial neural network memory controller 120 according to the examples of this disclosure is not limited to methods for monitoring data access requests.

[0148] The artificial neural network memory controller 120 can be configured to further include any memory for recording or monitoring data access requests. However, the artificial neural network memory controller 120 according to the example of this disclosure is not limited thereto and can be configured to communicate with external memory.

[0149] The artificial neural network memory controller 120 can be configured to monitor or record data access requests received from the processor 110 to analyze the data access requests.

[0150] In other words, the artificial neural network memory controller 120 can be configured to analyze received data access requests to analyze the artificial neural network data locality of the artificial neural network model being processed by the processor 110.

[0151] In other words, the artificial neural network memory controller 120 can be configured to analyze the artificial neural network data locality of an artificial neural network model compiled to operate at the processor-memory level.

[0152] In other words, the artificial neural network memory controller 120 can be configured to analyze the artificial neural network data locality at the processor-memory level based on the artificial neural network model, and to analyze the operation processing order of the artificial neural network in the unit of the memory access request generated by the processor, so as to analyze the artificial neural network data locality of the artificial neural network model.

[0153] Based on the above configuration, the artificial neural network memory controller 120 can analyze the locality of artificial neural network data reconstructed at the processor-memory level.

[0154] In some examples, the compiler can be configured to analyze the artificial neural network data locality of the artificial neural network model in word units.

[0155] In some examples, at least one artificial neural network memory controller can be configured to provide artificial neural network data locality analyzed by the compiler in word units. Here, the word unit can vary from 8 bits, 16 bits, 32 bits, 64 bits, etc., depending on the word unit of processor 110. Here, the word unit can be set to different word units, such as 2 bits, 3 bits, or 5 bits, depending on the quantization algorithm of the kernel, feature maps, etc. of the compiled artificial neural network model.

[0156] The artificial neural network memory controller 120 can be configured to include special function registers. These special function registers can be configured to store locality information of the artificial neural network data.

[0157] The artificial neural network memory controller 120 can be configured to operate in different modes depending on whether locality information of the artificial neural network data is stored.

[0158] If the artificial neural network memory controller 120 stores artificial neural network data locality information, it can predict in advance the data processing order of the artificial neural network model to be processed by the processor 110 in word unit order, allowing the controller to be configured not to record individual data access requests. However, this is not the only limitation, and the controller can be configured to verify whether there are errors in the stored artificial neural network data locality when comparing the stored information with data access requests generated by the processor.

[0159] If the artificial neural network memory controller 120 is not provided with artificial neural network data locality information, the artificial neural network memory controller 120 may be configured to observe data access requests generated by the processor 110 to operate in a mode in which the artificial neural network data locality of the artificial neural network model processed by the processor 110 is predicted.

[0160] In some examples, the artificial neural network memory system can be configured to include a processor, memory, and cache, and to generate in advance predicted data access requests, including data to be requested by the processor, based on artificial neural network data locality information. The predicted data access request can be referred to as an ANN DL-based predicted data access request, or a data access request to be requested. For the convenience of the following description, the predicted data access request can be referred to as a pre-prepared data access request. The artificial neural network memory system can be configured to store data corresponding to the predicted data access request from the memory in the cache before the processor requests it. In this case, the artificial neural network memory system can be configured to operate in either a first mode configured to operate by receiving artificial neural network data locality information or a second mode configured to operate by observing data access requests generated by the processor to predict artificial neural network data locality information. According to the above configuration, when the artificial neural network memory system is provided with artificial neural network data locality information, the data to be requested by the processor is predicted and prepared in advance in word units. Furthermore, even if artificial neural network data locality information is not provided, the data access requests generated by the processor are monitored for a predetermined period of time to predict the artificial neural network data locality being processed by the processor in the data access request unit. Furthermore, even when artificial neural network (ANN) data locality information is provided, the ANN memory system independently monitors data access requests to reconstruct the ANN data locality and verify the provided ANN data locality. Therefore, changes or errors in the ANN model can be sensed.

[0161] In some examples, at least one artificial neural network memory controller and at least one processor can be configured to communicate directly with each other. According to the above configuration, the artificial neural network memory controller can directly receive data access requests from the processor, thereby eliminating latency caused by the system bus between the processor and the artificial neural network memory controller. In other words, direct communication between the processor and the artificial neural network memory controller may also include a dedicated bus or a dedicated communication channel, but this disclosure is not limited thereto.

[0162] In some examples, the locality information of artificial neural network data can be configured to be selectively stored in processor 110 and / or artificial neural network memory controller 120. The locality information of artificial neural network data can also be configured to be stored in special function registers included in processor 110 and / or artificial neural network memory controller 120. However, this is not a limitation, and the locality information of artificial neural network data can be configured to be stored in any memory or register that can communicate with the artificial neural network memory system.

[0163] Figure 2 The examples in this disclosure illustrate patterns of data locality in artificial neural networks. Reference will be made below. Figure 2 Describe the data locality and data locality patterns of artificial neural network models.

[0164] The artificial neural network memory controller 120 is configured to sequentially record or monitor data access requests received from the processor 110.

[0165] The artificial neural network memory controller 120 is configured to generate artificial neural network data locality patterns, which include the data locality of the artificial neural network model being processed by the processor 110. Specifically, the artificial neural network memory controller 120 can be configured to analyze data access requests associated with the artificial neural network model generated by the processor 110 to generate recurring specific patterns. In other words, when a data access request is observed, artificial neural network data locality information can be stored as an artificial neural network data locality pattern.

[0166] refer to Figure 2 For example, the artificial neural network memory controller 120 sequentially records eighteen data access requests. Each data access request is configured to include identification information.

[0167] The identification information included in the data access request can be configured to include various types of information.

[0168] For example, the identification information can be configured to include at least a memory address value and an operating mode value.

[0169] For example, the memory address value can be configured to include the memory address value corresponding to the requested data, but this disclosure is not limited thereto.

[0170] For example, the memory address value can be configured to include a start value and an end value for the memory address corresponding to the requested data. Based on this configuration, data is considered to be stored sequentially between the start and end values ​​of the memory address. Therefore, the capacity used to store the memory address value can be reduced. That is, the memory can also operate in burst mode when a trigger value is activated.

[0171] For example, the memory address value can be configured to include a starting value for the memory address corresponding to the requested data and a data sequential read trigger value. According to this configuration, data can be read sequentially from the starting value of the memory address until the sequential read trigger value changes. This configuration allows for continuous data reading, thereby increasing the effective memory bandwidth.

[0172] For example, the memory address value can be configured to include a starting value of the memory address corresponding to the requested data and information about the amount of data. The number of units containing the amount of data can be determined based on the memory capacity. For example, a unit can be one of an 8-bit byte, a 4-byte word, or a 1024-byte word block, but this disclosure is not limited thereto. According to the above configuration, data can be read continuously from the starting value of the memory address, which is equal to the amount of data in the set unit size, thereby increasing the effective memory bandwidth.

[0173] For example, when the memory is non-volatile, the memory address value may also include a physical logical address mapping table or flash translation layer information, but this disclosure is not limited thereto.

[0174] For example, the operating mode can be configured to include both read and write modes. Read and write operations can also include burst modes.

[0175] For example, the operating mode can be configured to further include overwrite, but this disclosure is not limited thereto.

[0176] The artificial neural network memory controller 120 can be configured to determine whether the identification information for each data access request is the same.

[0177] For example, the artificial neural network memory controller 120 can be configured to determine whether the memory address and operating mode of each data access request are the same. In other words, the artificial neural network memory controller 120 can be configured to detect data access request values ​​with the same memory address value and the same operating mode.

[0178] For example, when the memory address value and operating mode of the first data access request are the same as those of the tenth data access request, the artificial neural network memory controller 120 is configured to generate an artificial neural network data locality pattern corresponding to the corresponding memory address value and operating mode.

[0179] The data locality pattern of artificial neural networks is configured to include data in which memory addresses are recorded sequentially for data access requests.

[0180] In other words, the artificial neural network memory controller 120 can be configured to detect recurring cycles of data access requests with the same memory address value and operating mode in order to generate an artificial neural network data locality pattern configured by data access requests with recurring memory address values ​​and operating modes.

[0181] In other words, the artificial neural network memory controller 120 can be configured to generate artificial neural network data locality patterns by detecting repetitive patterns of memory addresses included in data access requests.

[0182] refer to Figure 2 When the artificial neural network memory controller 120 identifies that the memory address value and operating mode of the first data access request are the same as those of the tenth data access request, the artificial neural network memory controller 120 can be configured to generate an artificial neural network data locality pattern from the predicted data access requests from the initial data access request to the repeated data access requests within the same data access requests. In this case, the artificial neural network memory controller 120 can be configured to generate artificial neural network data locality patterns including the first data access request to the ninth data access request.

[0183] In other words, reference Figure 2 The described artificial neural network data locality pattern can be configured to include memory address values ​​and operational model values ​​in the order of first data access request, second data access request, third data access request, fourth data access request, fifth data access request, sixth data access request, seventh data access request, eighth data access request, and ninth data access request.

[0184] The artificial neural network data locality patterns generated by the artificial neural network memory controller 120 can be stored in various forms, such as log files, tables, or lists. The artificial neural network memory controller 120 according to the examples of this disclosure is not limited to the recording type or format of the artificial neural network data locality patterns.

[0185] The artificial neural network data locality patterns generated by the artificial neural network memory controller 120 can be stored in any memory of the artificial neural network memory controller 120. The artificial neural network memory controller 120 according to the examples of this disclosure is not limited to the structure or method of storing the memory for artificial neural network data locality patterns.

[0186] The artificial neural network memory controller 120 may be configured to further include arbitrary memory for storing locality patterns of artificial neural network data. However, the artificial neural network memory controller 120 according to the examples of this disclosure is not limited thereto and may be configured to communicate with external memory.

[0187] In other words, the artificial neural network memory system 100 according to the example of this disclosure can be configured to include at least one processor 110 and an artificial neural network memory controller 120, wherein the at least one processor 110 is configured to generate data access requests corresponding to artificial neural network operations, and the artificial neural network memory controller 120 is configured to sequentially record data access requests to generate artificial neural network data locality patterns.

[0188] When the artificial neural network memory controller 120 generates an artificial neural network data locality pattern, the artificial neural network memory controller 120 can be configured to determine whether the memory address value and operation mode value of the data access request received from the processor 110 match any of the memory address value and operation mode value included in the previously generated artificial neural network data locality pattern.

[0189] refer to Figure 2 When the artificial neural network memory controller 120 receives a tenth data access request from the processor 110, the artificial neural network memory controller 120 can be configured to determine whether the received data access request has the same memory address value as the memory address value contained in the artificial neural network data locality pattern.

[0190] refer to Figure 2 When the artificial neural network memory controller 120 receives a tenth data access request, the artificial neural network memory controller 120 can be configured to detect that the start value [0] and end value [0×1000000] of the memory address value of the tenth data access request are the same as the start and end memory address values ​​of the first data access request, and can be configured to detect that the read mode value of the operation mode of the tenth data access request is the same as the read mode value of the operation mode of the first data access request. Therefore, the artificial neural network memory controller 120 determines that the tenth data access request is the same as the first data access request, and that the tenth data access request is an artificial neural network operation.

[0191] When the artificial neural network memory controller 120 receives an eleventh data access request, it can be configured to detect that the start value [0x1100000] and end value [0x1110000] of the memory address value of the eleventh data access request are the same as the start and end memory address values ​​of the second data access request, and can also be configured to detect that the write mode value of the operation mode of the eleventh data access request is the same as the write mode value of the operation mode of the second data access request. Therefore, the artificial neural network memory controller 120 determines that the eleventh data access request is the same as the second data access request, and that the eleventh data access request is an artificial neural network operation.

[0192] In other words, the artificial neural network memory controller 120 can distinguish the start and end of an artificial neural network data locality pattern. Furthermore, even without a specific command after the end of an artificial neural network data locality pattern, the artificial neural network memory controller 120 can prepare for the start of the next artificial neural network data locality pattern in advance. Therefore, when repeating the same operation, by predicting the start of the next inference based on the end of the current inference, it has the effect of preparing data before the start of the next inference. Thus, when repeating the same artificial neural network data locality pattern, start and end delay times can be prevented or reduced.

[0193] Refer again Figure 2 The artificial neural network memory controller 120 does not generate an artificial neural network data locality pattern from the first data access request to the ninth data access request. In this case, the artificial neural network memory controller 120 is initialized, or the processor 110 does not perform artificial neural network operations. Therefore, the artificial neural network memory controller 120 does not detect a pattern match with the ninth data access request. The artificial neural network memory controller 120 can determine the identity of the first data access request on the tenth data access request, generate an artificial neural network data locality pattern, and record whether the pattern matches. The tenth to eighteenth data access requests are the same as the first to ninth data access requests, so the artificial neural network memory controller 120 can determine that the patterns of the tenth to eighteenth data access requests match the artificial neural network data locality pattern.

[0194] In other words, the artificial neural network memory controller 120 can be configured to determine whether the operation being processed by the processor 110 is an artificial neural network operation by utilizing artificial neural network data locality patterns. According to this configuration, even if the artificial neural network memory controller 120 only receives a data access request including a memory address value and an operation mode value generated by the processor 110, the artificial neural network memory controller 120 can determine that the processor 110 is processing an artificial neural network operation. Therefore, the artificial neural network memory controller 120 can determine whether the processor 110 is currently executing an artificial neural network operation based on artificial neural network data locality patterns without requiring separate additional identification information.

[0195] As will refer to Figure 2 As described further, each data access request can be configured to be stored as a token. For example, each data access request for an artificial neural network can be tokenized for storage. For example, each data access request for an artificial neural network can be tokenized based on identification information. For example, each data access request for an artificial neural network can be tokenized based on a memory address value. However, the examples in this disclosure are not limited thereto, and the token can be referred to as a code, identifier, etc. For example, tokens can be defined by word, data access request, or ANN DL.

[0196] For example, a first data access request can be stored as a token [1]. A fourth data access request can be stored as a token [4]. A seventh data access request can be stored as a token [7]. For example, an artificial neural network data locality pattern can be stored as a token [1-2-3-4-5-6-7-8-9]. For example, a tenth data access request has the same memory address value and the same operation mode value as the token [1], so the tenth data access request can be stored as a token [1]. A thirteenth data access request has the same memory address value and the same operation mode value as the token [4], so the thirteenth data access request can be stored as a token [4]. Therefore, when the artificial neural network memory controller 120 detects a token that is the same as the token of the artificial neural network data locality pattern, the artificial neural network memory controller can be configured to determine that the corresponding data access request is an artificial neural network operation.

[0197] Based on the above configuration, the artificial neural network memory controller 120 can easily and quickly identify and distinguish data access requests by utilizing tokenized artificial neural network data locality patterns. Furthermore, even when additional identification information and / or data are added to the data access request, the artificial neural network memory controller uses the same token to utilize the token, even when the additional information in the data access request increases to the point that the data access request can be easily and quickly identified and distinguished.

[0198] In some examples, artificial neural network data locality patterns stored in the artificial neural network memory controller can be eliminated or initialized. For example, before an artificial neural network data locality pattern is used when it expires at a predetermined time, such as when no data access request matching the artificial neural network data locality pattern is generated at a specific time, the artificial neural network memory controller determines that the artificial neural network data locality pattern is used infrequently, and eliminates or initializes the artificial neural network data locality pattern.

[0199] Based on the above configuration, the availability of storage space in the memory storing locality patterns of artificial neural network data can be improved.

[0200] In some examples, the artificial neural network memory controller can be configured to store updated and previous patterns of artificial neural network data locality patterns to determine whether the artificial neural network model has changed. That is, when multiple artificial neural network models exist, the artificial neural network memory controller can be configured to further generate artificial neural network data locality patterns corresponding to the number of artificial neural network models.

[0201] For example, when the first artificial neural network data locality mode is token [1-2-3-4-5-6-7-8-9] and the second artificial neural network data locality mode is token [11-12-13-14-15-16-17-18], if the processor generates a data access request corresponding to token [1], the artificial neural network memory controller can be configured to select the first artificial neural network data locality mode. Alternatively, if the processor generates a data access request corresponding to token

[11] , the artificial neural network memory controller can be configured to select the second artificial neural network data locality mode.

[0202] According to the above configuration, the artificial neural network memory controller can store multiple artificial neural network data locality patterns, and when the artificial neural network model processed by the processor is changed to another artificial neural network model, the previously stored artificial neural network data locality patterns can be quickly applied.

[0203] In some examples, the artificial neural network memory controller can be configured to determine whether a data access request is a request from one artificial neural network model or a mixture of requests from multiple artificial neural network models. Furthermore, the artificial neural network memory controller can be configured to predict data access requests corresponding to the artificial neural network data locality of each of the multiple artificial neural network models.

[0204] For example, a processor can process multiple artificial neural network models simultaneously, and in this case, the data access request generated by the processor can be a mixed data access request corresponding to multiple artificial neural network models.

[0205] For example, when the first artificial neural network data locality pattern is the token [1-2-3-4-5-6-7-8-9] and the second artificial neural network data locality pattern is the token [11-12-13-14-15-16-17-18], the processor 110 can generate tokens corresponding to the data access request in the order of [1-11-2-3-12-13-14-4-5-6-15-16-7-8-9].

[0206] The artificial neural network memory controller knows the locality pattern of each artificial neural network data, so that even if token [1] is generated, and then token

[11] is generated, the artificial neural network memory controller can predict that token [2] will be generated next. Therefore, the artificial neural network memory controller can generate a predicted data access request corresponding to token [2] in advance. Furthermore, even if token [2] is generated after token

[11] is generated, the artificial neural network memory controller can predict that token

[12] will be generated next. Therefore, the artificial neural network memory controller can generate a predicted data access request corresponding to token

[12] in advance.

[0207] Based on the above configuration, the artificial neural network memory controller 120 predicts data access requests that will be generated by the processor 110, which processes multiple artificial neural network models, for each artificial neural network model, in order to predict and prepare the data that will be requested by the processor 110.

[0208] In some examples, the artificial neural network memory controller can be configured to store multiple artificial neural network data locality patterns.

[0209] For example, when a processor processes two artificial neural network models, the artificial neural network memory controller can be configured to store the artificial neural network data locality patterns of each artificial neural network model.

[0210] Based on the above configuration, when processing the operation of each artificial neural network model, the actual data access request corresponding to each model can be predicted, thereby potentially improving the processing speed of artificial neural network operations according to the examples of this disclosure.

[0211] In some examples, the artificial neural network memory controller can be configured to further include an artificial neural network model, which is configured to machine learning artificial neural network data locality patterns.

[0212] Based on the above configuration, the artificial neural network (ANN) model of the ANN memory controller can be configured to perform reinforcement learning in real time in response to data access requests generated by the processor. Furthermore, the ANN model of the ANN memory controller can be a model trained using known ANN data locality patterns as learning data. Therefore, the ANN memory controller can extract ANN data locality patterns from various ANN models. Specifically, this approach may be effective when multiple ANN models are processed by requests from multiple users (e.g., servers).

[0213] As will refer to Figure 2 As described elsewhere, the artificial neural network memory controller 120 can be configured to dynamically and in real time monitor the artificial neural network model processed by the processor 110 and determine whether the artificial neural network model has changed.

[0214] For example, the artificial neural network memory controller 120 can be configured to statistically determine the reliability of the artificial neural network data locality patterns using the pattern matching frequency of the artificial neural network data locality patterns. It can be configured such that the reliability of the artificial neural network data locality patterns increases as the pattern matching frequency increases, and the reliability of the artificial neural network data locality patterns decreases as the pattern matching frequency decreases.

[0215] According to the above configuration, when the processor 110 repeatedly processes a specific artificial neural network model, the artificial neural network memory controller 120 can improve the predictive reliability of the artificial neural network data locality of the specific artificial neural network model.

[0216] Figure 3 This is an exemplary artificial neural network model illustrating patterns of data locality in artificial neural networks applicable to various examples of this disclosure.

[0217] Currently being made by Figure 3 The exemplary artificial neural network model 1300 processed by the processor 110 shown can be any artificial neural network model trained to perform a specific reasoning function. For the sake of description, an artificial neural network model in which all nodes are fully connected has been shown, but this disclosure is not limited thereto.

[0218] although Figure 3Not shown, but applicable to the artificial neural network model of this disclosure, may be a convolutional neural network (CNN), which is a type of deep neural network (DNN). Exemplary artificial neural network models may be, for example, fully convolutional networks (FCNs) with VGG, VGG16, DenseNET and encoder-decoder structures, deep neural networks (DNNs) such as SegNet, DeconvNet, DeepLABV3+, or U-net, or SqueezeNet, AlexNet, ResNet18, MobileNet-v2, GoogLeNet, ResNet-v2, ResNet50, ResNet101 and Inception-v3, or ensemble models based on at least two different models, but the artificial neural network models of this disclosure are not limited thereto.

[0219] The exemplary artificial neural network model described above can be configured to have artificial neural network data locality.

[0220] Refer again Figure 3 The artificial neural network data locality of the artificial neural network model processed by processor 110 is described in detail.

[0221] An exemplary artificial neural network model 1300 includes an input layer 1310, a first connection network 1320, a first hidden layer 1330, a second connection network 1340, a second hidden layer 1350, a third connection network 1360, and an output layer 1370.

[0222] Artificial neural networks have corresponding weights. The weights of the connection network are multiplied by the input node values, and the accumulated multiplication values ​​are stored in the corresponding output layer nodes.

[0223] In other words, the connection network of the artificial neural network model 1300 is represented by lines, while the weights are represented by symbols. express.

[0224] In other words, various activation functions can be additionally provided to impart nonlinearity to the accumulated values. Activation functions may be, for example, the sigmoid function, hyperbolic tangent function, ELU function, Hard-Sigmoid function, Swish function, Hard-Swish function, SELU function, CELU function, GELU function, TANHSHRINK function, SOFTPLUS function, MISH function, piecewise interpolation approximation of a nonlinear function, or ReLU function, but this disclosure is not limited thereto.

[0225] The input layer 1310 of the exemplary artificial neural network model 1300 includes input nodes x1 and x2.

[0226] The exemplary artificial neural network model 1300 includes a first connection network 1320 with six weight values, which connects the nodes of the input layer 1310 and the nodes of the first hidden layer 1330.

[0227] The first hidden layer 1330 of the exemplary artificial neural network model 1300 includes nodes a1, a2, and a3. The weight values ​​of the first connection network 1320 are multiplied by the node values ​​of the corresponding input layer 1310, and the accumulated value of the multiplication is stored in the first hidden layer 1330.

[0228] The second connection network 1340 of the exemplary artificial neural network model 1300 includes a connection network with nine weight values ​​that connects the nodes of the first hidden layer 1330 and the nodes of the second hidden layer 1350.

[0229] The second hidden layer 1350 of the exemplary artificial neural network model 1300 includes nodes b1, b2, and b3. The weight values ​​of the second connection network 1340 are multiplied by the corresponding node values ​​of the first hidden layer 1330, and the accumulated value of the multiplication is stored in the second hidden layer 1350.

[0230] The exemplary artificial neural network model 1300 includes a third connection network 1360 with six weight values, which connects the nodes of the second hidden layer 1350 and the nodes of the output layer 1370.

[0231] The output layer 1370 of the exemplary artificial neural network model 1300 includes nodes y1 and y2. The weight values ​​of the third connection network 1360 are multiplied by the corresponding input node values ​​of the second hidden layer 1350, and the accumulated value of the multiplication is stored in the output layer 1370.

[0232] Based on the structure of the aforementioned artificial neural network model 1300, it is recognized that the operations of each layer need to be executed sequentially. In other words, a potential problem is that once the structure of the artificial neural network model is determined, the order of operations for each layer needs to be determined, and the inference results may be inaccurate when operations are performed in different orders. The order of operations or the order of data flow based on the structure of the artificial neural network model can be defined as data locality in artificial neural networks.

[0233] Furthermore, for ease of description, in Figure 2 Although layer units are described herein, the examples of this disclosure are not limited to layer units. The processor 110 according to the examples of this disclosure processes data based on the data locality of an artificial neural network, enabling the processor to operate at word units or data access request units, rather than at layer units. Here, the data size of the data access request can be less than or equal to the data size of the corresponding layer.

[0234] Refer again Figure 3 For example, for the multiplication operation of the weight value of the first connection network 1320 and the node value of the input layer 1310, the processor 110 can generate a data access request at the layer unit level.

[0235] However, the layer operations of the weight values ​​of the first connection network 1320 and the node values ​​of the input layer 1310 are not processed as a single data access request. Instead, they are processed as multiple partitioned sequential data access requests based on the feature map of the processor 110, the smoothing technique of the processing element, the number of processing elements of the processor, the cache capacity of the processor 110, the memory hierarchy of the processor 110, and / or the compiler algorithm of the processor 110.

[0236] When a data access request to be requested by processor 110 is divided into multiple data access requests, the order in which the data access requests are divided can be determined by artificial neural network data locality. In this case, artificial neural network memory controller 120 can be configured to have the artificial neural network data locality to be prepared to provide data corresponding to the actual data access request to be requested by processor 110. The “actual data access request” can be referred to as “the data access request generated by the processor after the predicted data access request.” Alternatively, artificial neural network memory controller 120 can be configured to predict the artificial neural network data locality to be prepared to provide data corresponding to subsequent data access requests to be requested by processor 110.

[0237] Will be in Figure 3 The data access requests generated by the processor 110 and the data locality of the artificial neural network during the operation of the artificial neural network model 1300 shown are described.

[0238] Processor 110 generates a first data access request to read the input node values ​​of the input layer 1310 of artificial neural network model 1300. The first data access request includes a first memory address value and a read mode value. The first data access request can be stored as a token [1].

[0239] Next, the processor 110 generates a second data access request to read the weight values ​​of the first connection network 1320 of the artificial neural network model 1300. The second data access request includes a second memory address value and a read mode value. The second data access request can be stored as a token [2].

[0240] Next, the processor 110 generates a third data access request for storing the node value of the first hidden layer 1330, which is obtained by multiplying and summing the weight values ​​of the first connection network 1320 of the artificial neural network model 1300 with the node values ​​of the input layer 1310. The third data access request includes a third memory address value and a write mode value. The third data access request can be stored as a token [3].

[0241] Next, processor 110 generates a fourth data access request to read node values ​​stored in the first hidden layer 1330 of the artificial neural network model 1300. The fourth data access request includes a third memory address value and a read mode value. The fourth data access request can be stored as a token [4].

[0242] Next, processor 110 generates a fifth data access request to read the weight values ​​of the second connection network 1340 of artificial neural network model 1300. The fifth data access request includes a fifth memory address value and a write mode value. The fifth data access request can be stored as a token [5].

[0243] Next, processor 110 generates a sixth data access request for storing node values ​​of the second hidden layer 1350, which are obtained by multiplying and summing the weight values ​​of the second connection network 1340 of the artificial neural network model 1300 with the node values ​​of the first hidden layer 1330. The sixth data access request includes a sixth memory address value and a write mode value. The sixth data access request may be stored as a token [6].

[0244] Next, processor 110 generates a seventh data access request to read node values ​​stored in the second hidden layer 1350 of artificial neural network model 1300. The seventh data access request includes a sixth memory address value and a read mode value. The seventh data access request can be stored as a token [7].

[0245] Next, processor 110 generates an eighth data access request to read the weight values ​​of the third connection network 1360 of artificial neural network model 1300. The eighth data access request includes an eighth memory address value and a read mode value. The eighth data access request can be stored as a token [8].

[0246] Next, processor 110 generates a ninth data access request to store the node values ​​of output layer 1370 obtained by multiplying and summing the weight values ​​of the third connection network 1360 of artificial neural network model 1300 with the node values ​​of the second hidden layer 1350. The ninth data access request includes a ninth memory address value and a write mode value. The ninth data access request can be stored as a token [9]. The node values ​​can be feature maps, activation maps, etc., but are not limited to these. The weight values ​​can be kernel windows, but are not limited to these.

[0247] In other words, processor 110 needs to generate first through ninth data access requests for the inference of the exemplary artificial neural network model 1300. If the sequence of data access requests generated by processor 110 is mixed, the artificial neural network data locality of artificial neural network model 1300 is violated, which may lead to errors in the inference results of artificial neural network model 1300 or affect accuracy. For example, processor 110 may compute the second layer first and then compute the first layer. Therefore, processor 110 can be configured to generate data access requests sequentially based on artificial neural network data locality. Thus, it is assumed that when processor 110 operates the artificial neural network, artificial neural network memory controller 120 can generate data access requests sequentially based on artificial neural network data locality.

[0248] However, as mentioned above, each data access request can be reinterpreted at the processor-memory level based on the processor's hardware characteristics. In the example above, it was described that the available capacity of the processor's cache memory was sufficient, and the data size of the node values ​​and weight values ​​was smaller than the available capacity of the cache memory. Therefore, processing each layer in a single data access request unit was described. If the data size of the artificial neural network model, such as weight values, feature maps, kernels, activation maps, etc., is larger than the available capacity of the processor's cache memory, the corresponding data access request may be divided into multiple data access requests, and in this case, the artificial neural network data locality of the artificial neural network model can be reconstructed.

[0249] The artificial neural network memory controller 120 according to the example of this disclosure can generate artificial neural network data locality patterns, such that the artificial neural network memory controller can operate to correspond to the artificial neural network data locality of an artificial neural network model to be actively processed by a processor.

[0250] In other words, even if the actual artificial neural network data locality of the artificial neural network model being processed by the processor 110 is unknown, the artificial neural network memory controller 120 can actually analyze the artificial neural network data locality by analyzing the recorded data access requests.

[0251] In other words, even without providing structural information about the artificial neural network model being processed by processor 110, artificial neural network memory controller 120 can actually analyze the data locality of artificial neural networks by analyzing recorded data access requests.

[0252] In some examples, the artificial neural network memory controller can be configured to have artificial neural network data locality patterns generated in advance at the processor-memory level.

[0253] Figure 4 An example according to this disclosure illustrates analysis by an artificial neural network memory controller. Figure 3 The artificial neural network data locality pattern 1400 was obtained from the artificial neural network model. Figure 5 The diagram illustrates the relationship between... Figure 4 The token and recognition information corresponding to the locality pattern of artificial neural network data are 1500. That is to say, Figure 5 The diagram illustrates the corresponding... Figure 4 The recognition information 1500 corresponding to the token of the artificial neural network data locality pattern 1400.

[0254] Figure 4 The artificial neural network data locality pattern 1400 shown is represented as a token for ease of description. (Reference) Figures 1A to 4 The artificial neural network data locality pattern 1400 of the artificial neural network model 1300 is stored as tokens [1-2-3-4-5-6-7-8-9].

[0255] Each data access request is configured to include identification information. Each data access request can be represented by a token, but this representation is merely for descriptive convenience. That is to say, this disclosure is not limited to tokens.

[0256] Based on the artificial neural network data locality pattern 1400, the artificial neural network memory controller 120 can sequentially predict the order in which tokens will be generated after the current token.

[0257] For example, the artificial neural network data locality mode 1400 can be configured to have a loop-type mode, where the order is from the last token to the first token, but this disclosure is not limited thereto.

[0258] For example, the data locality pattern 1400 of an artificial neural network may be configured by a memory address with repetitive loop characteristics, but this disclosure is not limited thereto.

[0259] For example, the artificial neural network data locality pattern 1400 may be configured to further include identification information for identifying the start and end of operations of the artificial neural network model, but this disclosure is not limited thereto.

[0260] For example, the start and end of the artificial neural network data locality pattern 1400 can be configured as a start token and an end token of the pattern, but this disclosure is not limited thereto.

[0261] According to the above configuration, when the processor 110 repeatedly infers a specific artificial neural network model, since the artificial neural network data locality mode 1400 is a cyclic mode, the start of the next inference can be predicted even if the current inference of the specific artificial neural network model ends.

[0262] For example, when an artificial neural network model recognizes image objects from a front-facing camera installed in an autonomous vehicle at a rate of 30 IPS (inference per second), the same inference is repeated continuously in a specific loop. Therefore, by utilizing the aforementioned recurrent artificial neural network data locality pattern, repeated data access requests can be predicted.

[0263] When the identification information is additionally described as an example, the tokens [3] and [4] of the artificial neural network data locality pattern 1400 have the same memory address value but different operating modes. Therefore, even though the memory address values ​​are the same, the operating modes are different, so the artificial neural network memory controller 120 can be configured to classify the third data access request and the fourth data access request as different tokens. However, the identification information of the example of this disclosure is not limited to the operating mode, but can be configured to predict the artificial neural network data locality pattern using only the memory address value.

[0264] The artificial neural network memory controller 120 can be configured to generate corresponding predicted data access requests (i.e., advance data access requests) based on the neural network data locality pattern 1400.

[0265] The artificial neural network memory controller 120 can be configured to generate corresponding predicted data access requests based on the artificial neural network data locality pattern 1400.

[0266] The artificial neural network memory controller 120 can be configured to generate predicted data access requests in advance based on the artificial neural network data locality pattern 1400.

[0267] According to the above configuration, when the processor 110 generates a specific data access request included in the artificial neural network data locality pattern 1400, the artificial neural network memory controller 120 can sequentially predict at least one data access request after the specific data access request. For example, when the processor 110 generates a token [1], the artificial neural network memory controller 120 can predict that a data access request corresponding to the token [2] will subsequently be generated. For example, when the processor 110 generates a token [3], the artificial neural network memory controller 120 can predict that a data access request corresponding to the token [4] will subsequently be generated. For example, when the processor 110 generates a token [1], the artificial neural network memory controller 120 can predict that corresponding data access requests will be generated in the order of the tokens [2-3-4-5-6-7-8-9].

[0268] In other words, when processor 110 processes multiple artificial neural network models, unpredictable data locality patterns may intervene between tokens of artificial neural network data locality patterns 1400. For example, a new token [4] may be interrupted after a token [2]. However, even in this case, artificial neural network memory controller 120 can predict and prepare processor 110 to generate a token [3] after the token [2].

[0269] For example, when processor 110 generates a token [9], artificial neural network memory controller 120 can predict when processor 110 generates a token [1].

[0270] Figure 6 The example illustrations of this disclosure illustrate the generation of 1600 predicted data access requests and actual (next) actual data access requests by an artificial neural network memory controller based on the data locality pattern of an artificial neural network.

[0271] According to an example of this disclosure, the artificial neural network memory controller 120 can be configured to utilize artificial neural network data locality patterns to predict actual data access requests to be subsequently requested by the processor 110 in advance to generate predicted data access requests.

[0272] Reference Figure 6A data access request token refers to a token corresponding to a data access request received by the artificial neural network memory controller 120 from the processor 110. A predicted data access request token is a token corresponding to a data access request obtained by the artificial neural network memory controller 120 predicting a subsequent data access request to be requested by the processor 110 based on the artificial neural network data locality pattern. An actual data access request token is a data access request token actually generated by the processor 110 immediately after the predicted data access request token is generated. The tokens described herein are merely illustrative examples; that is, this disclosure is not limited to these tokens.

[0273] Data access requests generated by the processor and predicted data access requests predicted by the artificial neural network memory controller before processor generation can correspond to specific data access request tokens. In this case, the data access request matching the specific data access request token and the predicted data access request can be configured to have the same memory address. That is, the data access request and the predicted data access request can be configured to include the same memory address.

[0274] For example, when the data access request token is [3] and the predicted data access request token is [3], the memory address value of each token can be the same. That is, the data access request and the predicted data access request can be configured to include the same operation mode value. For example, when the data access request token is [3] and the predicted data access request token is [3], the operation mode value of each token can be the same.

[0275] refer to Figure 6 When processor 110 generates a data access request corresponding to token [1], artificial neural network memory controller 120 generates a predicted data access request corresponding to token [2]. After generating the predicted data access request, processor 110 generates a subsequent (actual) data access request corresponding to token [2]. Artificial neural network memory controller 120 is configured to determine whether the predicted data access request accurately predicts the actual data access request. The same token corresponds to both the predicted and actual data access requests, thus allowing artificial neural network memory controller 120 to determine a pattern match.

[0276] Next, for example, when processor 110 generates a data access request corresponding to token [2], artificial neural network memory controller 120 generates a predicted data access request corresponding to token [3]. After generating the predicted data access request, processor 110 generates a subsequent (actual) data access request corresponding to token [3]. Artificial neural network memory controller 120 is configured to determine whether the predicted data access request accurately predicts the subsequent (actual) data access request. The same token corresponds to both the predicted data access request and the subsequent (actual) data access request, allowing artificial neural network memory controller 120 to determine a pattern match.

[0277] For example, when processor 110 generates a data access request corresponding to token [9], artificial neural network memory controller 120 generates a predicted data access request corresponding to token [1]. After generating the predicted data access request, processor 110 generates a subsequent (actual) data access request corresponding to token [9]. Artificial neural network memory controller 120 is configured to determine whether the predicted data access request accurately predicts the subsequent (actual) data access request. The same token corresponds to both the predicted data access request and the subsequent (actual) data access request, thus allowing artificial neural network memory controller 120 to determine a pattern match.

[0278] When the processor 110 generates a subsequent (actual) data access request after the artificial neural network memory controller 120 generates a predicted data access request, the artificial neural network memory controller 120 can be configured to determine whether the predicted data access request and the subsequent (actual) data access request are the same request.

[0279] Based on the above configuration, the artificial neural network memory system 100 can detect changes in the locality of artificial neural network data in the artificial neural network model processed by the processor 110. Therefore, even if the artificial neural network model changes, the artificial neural network memory controller 120 can analyze the changed locality of artificial neural network data.

[0280] When the artificial neural network memory controller 120 determines that the predicted data access request and the subsequent (actual) data access request are the same request, the artificial neural network memory controller 120 can be configured to maintain the artificial neural network data locality mode.

[0281] According to the above configuration, the artificial neural network memory system 100 detects that the artificial neural network model processed by the processor 110 is reused to prepare or provide data requested by the processor 110 more quickly.

[0282] When the artificial neural network memory controller 120 determines that the predicted data access request is different from the subsequent (actual) data access request, the artificial neural network memory controller 120 can be configured to update the artificial neural network data locality pattern or further generate a new artificial neural network data locality pattern.

[0283] According to the above configuration, the artificial neural network memory system 100 can detect changes in the artificial neural network model processed by the processor 110 to generate a predicted data access request corresponding to the changed artificial neural network model.

[0284] In some examples, the artificial neural network memory controller can be configured to generate continuous prediction data access requests.

[0285] For example, when the data access request token is [2], the predicted data access request generated by the artificial neural network memory controller can be the data access request corresponding to token [3]. However, it is not limited to this; for example, the predicted data access request generated by the artificial neural network memory controller can be multiple data access requests corresponding to tokens [3-4]. However, it is not limited to this; for example, the predicted data access request generated by the artificial neural network memory controller can be multiple data access requests corresponding to tokens [3-4-5-6].

[0286] Based on the above configuration, the artificial neural network memory controller can generate predicted data access requests based on the data locality pattern of the artificial neural network, predicting the entire sequence of continuously repeating data access requests.

[0287] Based on the above configuration, the artificial neural network memory controller can generate predicted data access requests that predict the order of at least some data access requests based on the data locality pattern of the artificial neural network.

[0288] Figure 7 The operation of an artificial neural network memory controller is illustrated in the example diagram of this disclosure.

[0289] refer to Figure 7 For artificial neural network operation processing, processor 110 can be configured to generate data access requests corresponding to the artificial neural network model based on the data locality of the artificial neural network.

[0290] The artificial neural network memory controller 120 sequentially records data access requests generated in the processor 110 to generate artificial neural network data locality patterns.

[0291] The artificial neural network memory controller 120 compares the generated artificial neural network data locality pattern with the data access request generated by the processor 110 to generate in advance a predicted data access request corresponding to a subsequent data access request to be generated by the processor 110.

[0292] An artificial neural network memory system 100 according to an example of this disclosure may be configured to include at least one processor 110, which is configured to generate data access requests corresponding to artificial neural network operations (S710), and may be further configured to generate artificial neural network data locality patterns of artificial neural network operations by sequentially recording data access requests (S720). The artificial neural network memory system 100 may also be configured to include at least one artificial neural network memory controller 120, which is configured to generate predicted data access requests based on artificial neural network data locality patterns to predict subsequent data access requests generated by the at least one processor 110.

[0293] In other words, at least one artificial neural network memory controller 120 generates a predicted data access request before generating a subsequent data access request (S730).

[0294] That is, at least one processor 110 is configured to send a data access request to at least one artificial neural network memory controller 120, and at least one artificial neural network memory controller 120 can be configured to output a predicted data access request corresponding to the data access request.

[0295] An artificial neural network memory system 100 according to an example of the present disclosure may be configured to include at least one processor 110 and at least one artificial neural network memory controller 120, the at least one processor 110 being configured to generate data access requests corresponding to artificial neural network operations, and the at least one artificial neural network memory controller 120 being configured to generate artificial neural network data locality patterns of artificial neural network operations by sequentially recording the data access requests generated by the at least one processor 110, and to generate predicted data access requests based on the artificial neural network data locality patterns that predict subsequent (actual) data access requests of the data access requests generated by the at least one processor 110.

[0296] Based on the above configuration, the artificial neural network memory controller 120 predicts subsequent (actual) data access requests generated by the artificial neural network model being processed by the processor 110 based on the data locality pattern of the artificial neural network. Advantageously, the corresponding data can be prepared in advance to be provided before the request from the processor 110.

[0297] The artificial neural network memory controller 120 can be configured to compare the generated predicted data access request with a subsequent data access request generated by the processor 110 after the predicted data access request is generated, in order to determine whether the artificial neural network data locality pattern matches (S740).

[0298] According to the above configuration, the artificial neural network memory controller 120 generates a predicted data access request before generating a subsequent data access request that will be prepared to provide data in advance. Therefore, the artificial neural network memory controller 120 can substantially eliminate or reduce the latency that may occur when data is provided to the processor 110.

[0299] Figure 8 An artificial neural network memory system 200 is illustrated according to another example of this disclosure.

[0300] refer to Figure 8 The artificial neural network memory system 200 can be configured to include a processor 210, an artificial neural network memory controller 220, and a memory 230.

[0301] Figure 8 Artificial neural network memory system 200 and Figure 1A The artificial neural network memory system 200 is basically the same as the artificial neural network memory system 100, except that the artificial neural network memory system 200 also includes a memory 230. Therefore, for the sake of convenience, redundant descriptions will be omitted.

[0302] The artificial neural network memory system 200 includes a memory 230 configured to communicate with an artificial neural network memory controller 220, and the memory 230 can be configured to operate according to a memory access request output from the artificial neural network memory controller 220.

[0303] Processor 210 can be configured to communicate with artificial neural network memory controller 220. Processor 210 can be configured to generate a data access request to be sent to artificial neural network memory controller 220. The data access request can be generated based on the artificial neural network data locality of the artificial neural network model being processed. Processor 210 is configured to be provided with data corresponding to the data access request from artificial neural network memory controller 220.

[0304] The artificial neural network memory controller 220 can be configured to receive data access requests generated by the processor 210. The artificial neural network memory controller 220 can also be configured to generate artificial neural network data locality patterns by analyzing the artificial neural network data locality of the artificial neural network model being processed by the processor 210.

[0305] The artificial neural network memory controller 220 can be configured to control the memory 230 by generating memory access requests. The artificial neural network memory controller 220 can be configured to generate memory access requests corresponding to data access requests. That is, the artificial neural network memory controller 220 can be configured to generate memory access requests corresponding to data access requests generated by the processor 210. For example, when the artificial neural network memory controller 220 does not generate artificial neural network data locality patterns, the artificial neural network memory controller 220 can be configured to generate memory access requests based on data access requests generated by the processor 210. In this case, the memory access request can be configured to include a memory address value and an operating mode value from the identification information contained in the data access request.

[0306] The artificial neural network memory controller 220 can be configured to generate memory access requests corresponding to predicted data access requests. That is, the artificial neural network memory controller 220 can be configured to generate memory access requests based on predicted data access requests generated from artificial neural network data locality patterns. For example, when the artificial neural network memory controller 220 generates artificial neural network data locality patterns, the artificial neural network memory controller 220 can be configured to generate memory access requests based on predicted data access requests.

[0307] According to the above configuration, the artificial neural network memory controller 220 can send data to and receive data from the memory 230 through memory access requests, and when a memory access request is generated based on a predicted data access request, the artificial neural network memory system 200 can provide data to the processor 210 more quickly.

[0308] The artificial neural network memory controller 220 can be configured to generate memory access requests based on either a data access request generated by the processor 210 or a predicted data access request generated by the artificial neural network memory controller 220. In other words, the memory access requests generated by the artificial neural network memory controller 220 can be selectively generated based on either a data access request or a predicted data access request.

[0309] The artificial neural network memory controller 220 can be configured to generate memory access requests that include at least a portion of the identification information included in the data access request and the predicted data access request. For example, the data access request generated by the processor 210 may include a memory address value and an operating mode value. In this case, the memory access request generated by the artificial neural network memory controller 220 can be configured to include the memory address value and operating mode value of the corresponding data access request.

[0310] In other words, each of the data access request, the predicted data access request, and the memory access request can be configured to include a corresponding memory address value and an operation mode value. The operation mode can be configured to include a read mode and a write mode. For example, a memory access request generated by the artificial neural network memory controller 220 can be configured to have the same data type as the data access request or the predicted data access request. Therefore, from the perspective of the memory 230, even without distinguishing between the data access request and the predicted data access request, the memory access request task can be executed according to the instructions of the artificial neural network memory controller 220.

[0311] Based on the above configuration, the memory 230 can operate regardless of whether the memory access request generated by the artificial neural network memory controller 220 is based on a data access request or a predicted data access request. Therefore, even if the artificial neural network memory controller 220 operates based on the data locality of artificial neural networks, the artificial neural network memory controller can operate in a way that is compatible with various types of memory.

[0312] The artificial neural network memory controller 220 sends a memory access request to the memory 230, and the memory 230 performs a memory operation corresponding to the memory access request.

[0313] The memory according to the examples of this disclosure can be implemented in various forms. The memory can be implemented as volatile memory and non-volatile memory.

[0314] Volatile memory may include dynamic RAM (DRAM) and static RAM (SRAM). Non-volatile memory may include programmable read-only memory (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, ferroelectric RAM (FRAM), magnetic RAM (MRAM), and phase-change memory devices (phase-change RAM), but this disclosure is not limited thereto.

[0315] The memory 230 can be configured to store at least one of inference data, weight data, and feature map data of the artificial neural network model being processed by the processor 210. The inference data may be the input signal of the artificial neural network model.

[0316] The memory 230 can be configured to receive memory access requests from the artificial neural network memory controller 220. The memory 230 can be configured to perform memory operations corresponding to the received memory access requests. The operating mode controlling the memory operations can include a read mode or a write mode.

[0317] For example, when the operating mode of the received memory access request is write mode, the memory 230 can store the data received from the artificial neural network memory controller 220 in the corresponding memory address value.

[0318] For example, when the received memory access request operates in read mode, memory 230 can transfer the data stored in the corresponding memory address value to artificial neural network memory controller 220. Artificial neural network memory controller 220 can be configured to send the received data to processor 210 again.

[0319] The memory 230 may have a delay. The delay of the memory 230 can refer to the delay time that occurs when the artificial neural network memory controller 220 processes the memory access request. That is, when the memory 230 receives a memory access request from the artificial neural network memory controller 220, the actual requested data is output from the memory 230 after a delay of a specific clock cycle.

[0320] To process a memory access request, memory 230 can access the memory address value included in the memory access request. Therefore, the time required to access the memory address value is necessary, and this time can be defined as memory latency. For example, the CAS latency of DDR4 SDRM memory is approximately 10 ns. When no data is supplied to processor 210 during the latency period, processor 210 is in an idle state, causing the processor not to perform any actual operations.

[0321] Furthermore, in the case of DRAM, which is a type of memory 230, multiple clock cycles are consumed to activate the word line and bit line according to the row address of the memory 230, multiple clock cycles are consumed to activate the column line, and multiple clock cycles are consumed to allow data to travel through the path traversed by the data being sent outside the memory 230. In the case of NAND flash memory, the cell activated at one time is large, allowing additional clock cycles to be consumed to search for data at the required address.

[0322] The memory 230 can have bandwidth. The data transfer rate of the memory 230 can be defined as the memory bandwidth. For example, the bandwidth of DDR4 SDRAM memory is approximately 4 GByte / sec. With higher memory bandwidth, the memory 230 can transfer data to the processor 210 faster.

[0323] In other words, the processing speed of the artificial neural network memory system 200 is affected by the latency generated when providing data to be processed by the processor 210 and the bandwidth performance of the memory 230, more than by the processing performance of the processor 210.

[0324] In other words, while memory bandwidth gradually increases, the improvement in memory latency is relatively slow compared to the rate of bandwidth improvement. Specifically, each memory access request incurs a memory latency of 230, making frequent memory access requests a significant reason for the slow speed of artificial neural processing.

[0325] In other words, even if the processor 210 operates at a high speed, if a delay occurs in obtaining the data required for the operation, the processor 210 may be in an idle state where it does not perform any operation. Therefore, in this case, the processing speed of the processor 210 will be slower.

[0326] Therefore, the artificial neural network memory system according to the example of this disclosure can be configured to increase the bandwidth and / or latency of memory 230.

[0327] Figure 9 The operation of the memory system is illustrated according to a comparative embodiment of this disclosure.

[0328] refer to Figure 9 The processor generates a data access request, and a known memory system can send a memory access request corresponding to the data access request to memory. At this point, the memory has a delay so that the requested data can be provided to the processor from memory after a waiting period.

[0329] For example, a known memory system receives a data access request [1] generated by a processor and sends a memory access request [1'] corresponding to the data access request [1]. The memory may send the data [1”] to the memory system after a delay. Therefore, the processor's processing time may be delayed by as much as the memory's delay with each data access request. Consequently, the inference operation time of the artificial neural network may be delayed by as much as the memory delay. Specifically, the artificial neural network inference operation time of a known memory system may be further delayed as the processor generates more data access requests.

[0330] Figure 10 The diagram illustrates the basis Figure 8 The operation of the memory system.

[0331] refer to Figure 10 The processor 210 generates a data access request [1], and the artificial neural network memory controller 220 can send a memory access request corresponding to the predicted data access request generated based on the data locality pattern of the artificial neural network to the memory 230. At this time, even if there is a delay in the memory 230, the processor 210 will generate a memory access request corresponding to the predicted data access request, so that when the processor 210 generates a subsequent data access request, the artificial neural network memory controller 220 can directly provide the data requested by the processor 210 to the processor 210.

[0332] For example, a data access request [1] generated by processor 210 is received by artificial neural network memory controller 220 to generate a predicted data access request [2] and a memory access request [2'] corresponding to the predicted data access request [2] is sent to memory 230. Memory 230 may send data [2”] to artificial neural network memory controller 220 after a delay. However, the data [2”] provided by memory 230 is the data corresponding to the memory access request [2'] based on the predicted data access request [2]. Therefore, when processor 210 generates a subsequent data access request [2], artificial neural network memory controller 220 may immediately provide data [2”] to processor 210.

[0333] If the time between a memory access request based on a predicted data access request and a subsequent data access request is longer than the latency of memory 230, then the artificial neural network memory controller 220 can provide data to the processor 210 once the subsequent data access request is received from the processor 210. In this case, the artificial neural network memory controller 220 can essentially eliminate the latency of memory 230.

[0334] In other words, when a memory access request based on a predicted data access request is sent to memory 230, the latency of memory 230 can be shorter than or equal to the time from the generation of the predicted data access request to the generation of a subsequent data access request. In this case, once processor 210 generates a subsequent data access request, the artificial neural network memory controller 220 can immediately provide the data without causing a delay.

[0335] Even if the time between a memory access request based on a predicted data access request and a subsequent data access request is shorter than the latency of memory 230, the latency of memory 230 will be significantly reduced, that is, reduced by as much as the time between the memory access request and the subsequent data access request.

[0336] According to the above configuration, the artificial neural network memory controller 220 can substantially eliminate or reduce the latency of the data to be provided to the processor 210.

[0337] In some examples, the artificial neural network memory controller of an artificial neural network memory system can be configured to measure the memory latency or be provided with a memory latency value from the memory.

[0338] Based on the above configuration, the artificial neural network memory controller can be configured to determine the timing of generating memory access requests based on predicted data access requests, according to memory latency. Therefore, the artificial neural network memory controller can generate memory access requests based on predicted data access requests that substantially minimize memory latency.

[0339] In some examples, the memory of an artificial neural network memory system can be a memory configured to include a refresh function that updates the voltage of memory elements (i.e., an array of memory elements). The artificial neural network memory controller can be configured to selectively control the refresh of memory address regions corresponding to memory access requests that correspond to predicted data access requests. For example, the memory can be a SAM or DRAM that includes a refresh function.

[0340] If the voltage of the memory cells in DRAM is not refreshed, the cells will slowly discharge, potentially resulting in data loss. Therefore, the voltage of the memory elements needs to be refreshed in each specific cycle. If the timing of memory access requests from the artificial neural network memory controller overlaps with the refresh timing, the artificial neural network memory system can be configured to advance or delay the timing of refreshing the voltage of the memory elements.

[0341] Artificial neural network memory systems can predict or compute the timing of memory access requests based on patterns of locality of data within artificial neural networks. Therefore, artificial neural network memory systems can be configured to limit the voltage refresh of memory elements during memory access request operations.

[0342] In other words, the inference operations of artificial neural networks operate on the concept of precision, so that even if some stored data is lost due to the delayed refresh of the memory element voltage, the reduction in inference precision may be negligible.

[0343] Based on the above configuration, data can be provided to the artificial neural network memory system according to memory access requests from the memory by adjusting the voltage refresh cycle of the memory elements. Therefore, the operating speed of the artificial neural network, which is reduced due to the voltage refresh of the memory elements, can be improved without significantly reducing inference accuracy.

[0344] In some examples, the memory of an artificial neural network memory system can be configured to further include a pre-charge function that charges the global bitline of the memory with a specific voltage. In this case, the artificial neural network memory controller can be configured to selectively provide pre-charge to memory address regions corresponding to memory access requests that correspond to predicted data access requests.

[0345] In some examples, an artificial neural network memory controller can be configured as a bitline of precharged or delayed memory that performs memory tasks corresponding to predicted data access requests based on patterns of data locality in artificial neural networks.

[0346] Typically, memory performs a precharge operation upon receiving memory access requests to execute read or write operations. When a memory operation completes, signals are left on the bit lines used for the data read / write operation and on each data input / output line, ensuring that subsequent memory operations can only proceed smoothly once these lines are precharged to a predetermined level. However, since precharge takes a considerable amount of time, the precharge time can delay memory operations when the timing of generating memory access requests overlaps with the timing of precharge. This, in turn, delays the processing time of data access requests requested by the processor.

[0347] Artificial neural network memory controllers can predict the execution order of memory operations on specific memory bit lines based on patterns of locality of data within artificial neural networks. Therefore, the artificial neural network memory controller can advance or delay the precharge time so that the precharge timing and the time for executing memory operations on specific bit lines do not overlap.

[0348] In other words, the inference operation of artificial neural network models operates on the concept of precision, so that even if some stored data is lost due to delayed precharging, the reduction in inference precision is negligible.

[0349] In other words, artificial neural networks are mathematical models that model the neural networks of biological systems, such as the brain. Human nerve cells, called neurons, exchange information through connections called synapses. This information exchange between nerve cells is very simple, but the aggregation of a large number of nerve cells creates intelligence. The advantage of this structure is that even if some nerve cells transmit incorrect information, it does not affect the overall information, making it highly robust to small errors. Therefore, due to these characteristics, even if the pre-charging and refresh functions of the memory storing the data of the artificial neural network model are selectively limited, the accuracy of the artificial neural network model will not be significantly affected, and memory latency caused by pre-charging or refreshing will be reduced.

[0350] Based on the above configuration, it is possible to improve the inference accuracy by reducing the operating speed of the pre-charged artificial neural network without substantially reducing the inference speed.

[0351] In some examples, the artificial neural network memory controller can be configured to independently control the memory's refresh and precharge functions based on the artificial neural network data locality pattern.

[0352] Figure 11 An artificial neural network memory system 300 is illustrated according to yet another example of this disclosure.

[0353] refer to Figure 11 The artificial neural network memory system 300 can be configured to include a processor 310, an artificial neural network memory controller 320 including a cache memory 322, and a memory 330.

[0354] The memory 330 included in the various examples of this disclosure may be a memory specifically designed for artificial neural network operations and may be referred to as a sequential access memory (SAM). However, this disclosure is not limited thereto, and the memory in the various examples of this disclosure may refer to a memory device specifically designed for artificial neural network operations that can be controlled based on the data locality of the artificial neural network.

[0355] The artificial neural network memory system 300 is basically the same as the artificial neural network memory system 200, except that the artificial neural network memory system 300 also includes a high-speed cache memory 322. Therefore, for the sake of convenience, redundant descriptions will be omitted.

[0356] The artificial neural network memory system 300 can be configured to include an artificial neural network memory controller 320, which includes a cache memory 322 configured to store data sent by the memory 330 in response to a memory access request based on a prediction-based data access request.

[0357] According to the above configuration, the artificial neural network memory controller 320 can read data and store it in the cache memory 322 in response to a memory access request based on a predicted data access request from the memory 330. Therefore, when the processor 310 generates a subsequent data access request, the artificial neural network memory controller 320 can immediately provide the data stored in the cache memory 322 to the processor 310.

[0358] The latency of cache memory 322 is much shorter than that of memory 330. The bandwidth of cache memory 322 is higher than that of memory 330.

[0359] The artificial neural network model processing performance of the artificial neural network memory system 300, which includes a high-speed cache memory 322, can be superior to that of the artificial neural network memory system 200.

[0360] Reference Figure 3 The artificial neural network model 1300 describes the artificial neural network memory system 300.

[0361] The artificial neural network model 1300 can be compiled by a specific compiler to operate in the processor 310. The compiler can be configured to provide the artificial neural network data locality pattern to the artificial neural network memory controller 320.

[0362] To infer the artificial neural network model 1300, the processor 310 can be configured to generate data access requests in sequence based on the data locality of the artificial neural network. Therefore, the artificial neural network memory controller 320 can monitor the data access requests to generate an artificial neural network data locality pattern 1400. Alternatively, the artificial neural network memory controller 320 can store the artificial neural network data locality pattern 1400 that has already been generated in advance.

[0363] The following section will describe an example of not generating artificial neural network data locality patterns 1400.

[0364] First, the processor 310 can generate a data access request for a token [1] corresponding to the node value read mode of the input layer 1310. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [1] to send the node value of the input layer 1310 from the memory 330 to the processor 310.

[0365] Next, the processor 310 can generate a data access request for a token [2] corresponding to the weight value of the first connection network 1320. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [2] to send the weight value of the first connection network 1320 sent from the memory 330 to the processor 310.

[0366] Next, the processor 310 receives the node values ​​of the input layer 1310 and the weight values ​​of the first connection network 1320 to calculate the node values ​​of the first hidden layer 1330. That is, the processor 310 can generate a data access request for a token [3] corresponding to the node value write pattern of the first hidden layer 1330. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [3] to store the node values ​​of the first hidden layer 1330 in the memory 330.

[0367] Next, the processor 310 can generate a data access request for a token [4] corresponding to the node value reading pattern of the first hidden layer 1330. Therefore, the artificial neural network memory controller 320 generates the memory access request for the token [4] to send the node value of the first hidden layer 1330 from the memory 330 to the processor 310.

[0368] Next, the processor 310 can generate a data access request for a token [5] corresponding to the weight value of the second connection network 1340. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [5] to send the weight value of the second connection network 1340 sent from the memory 330 to the processor 310.

[0369] Next, the processor 310 receives the node values ​​of the first hidden layer 1330 and the weight values ​​of the second connection network 1340 to calculate the node values ​​of the second hidden layer 1350. That is, the processor 310 can generate a data access request for a token [6] corresponding to the node value write pattern of the second hidden layer 1350. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [6] to store the node values ​​of the second hidden layer 1350 in the memory 330.

[0370] Next, the processor 310 can generate a data access request for a token [7] corresponding to the node value reading pattern of the second hidden layer 1350. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [7] to send the node value of the second hidden layer 1350 from the memory 330 to the processor 310.

[0371] Next, the processor 310 can generate a data access request for a token [8] corresponding to the weight value of the third connection network 1360. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [8] to send the weight value of the third connection network 1360 sent from the memory 330 to the processor 310.

[0372] Next, the processor 310 receives the node values ​​of the second hidden layer 1350 and the weight values ​​of the third connection network 1360 to calculate the node values ​​of the output layer 1370. That is, the processor 310 can generate a data access request for a token [9] corresponding to the node value write pattern of the output layer 1370. Therefore, the artificial neural network memory controller 320 generates a memory access request for the token [9] to store the node values ​​of the output layer 1370 in the memory 330.

[0373] Therefore, the artificial neural network memory system 300 can store the inference results of the artificial neural network model 1300 in the output layer 1370.

[0374] In the example above, the artificial neural network data locality pattern 1400 has not yet been generated in the artificial neural network memory controller 320. Therefore, according to the example above, a predicted data access request cannot be generated. Consequently, since the artificial neural network memory controller 320 does not provide data in advance, a delay in memory 330 may occur with each memory access request.

[0375] However, since the artificial neural network memory controller 320 records the data access request, when the processor 310 generates a data access request for the token [1] corresponding to the node value read pattern of the input layer 1310 again, the artificial neural network data locality pattern 1400 can be generated.

[0376] In the following text, see references Figure 4 The generation of the data locality pattern 1400 in artificial neural networks is described.

[0377] In the following example, an artificial neural network data locality pattern 1400 is generated and the processor 310 repeatedly infers the artificial neural network model 1300, but this disclosure is not limited thereto.

[0378] Processor 310 detects duplicate data access requests for token [1] to generate artificial neural network data locality pattern 1400. In other words, since artificial neural network memory controller 320 sequentially stores token [1] into token [9], artificial neural network data locality can be determined when artificial neural network memory controller 320 detects token [1] again.

[0379] However, as stated above, the artificial neural network memory controller according to the examples of this disclosure is not limited to tokens. Tokens are used for descriptive convenience only, and the examples of this disclosure can be implemented by including identification information in data access requests and memory access requests.

[0380] For example, when processor 310 generates a data access request corresponding to token [9], artificial neural network memory controller 320 generates a predicted data access request for token [1]. Accordingly, artificial neural network memory controller 320 generates a memory access request for token [1] to pre-store the node values ​​of input layer 1310 in cache memory 322.

[0381] In other words, if the data access request for token [9] is the last step of the artificial neural network model 1300, the artificial neural network memory controller 320 can predict that the data access request for token [1], which is the starting step of the artificial neural network model 1300, will be generated.

[0382] Next, when the processor 310 generates a data access request for token [1], the artificial neural network memory controller 320 determines whether the predicted data access request for token [1] is the same as the data access request for token [1]. When it is determined that the requests are the same, the node value of the input layer 1310 stored in the cache memory 322 can be immediately provided to the processor 310.

[0383] At this time, the artificial neural network memory controller 320 generates a predicted data access request for a token [2].

[0384] Accordingly, the artificial neural network memory controller 320 generates a memory access request for a token [2] to pre-store the weight values ​​of the first connection network 1320 in the cache memory 322.

[0385] Next, when the processor 310 generates a data access request for token [2], the artificial neural network memory controller 320 determines whether the predicted data access request for token [2] is the same as the data access request for token [2]. When it is determined that the requests are the same, the node values ​​of the first connection network 1320 stored in the cache memory 322 can be immediately provided to the processor 310.

[0386] At this time, the artificial neural network memory controller 320 generates a predicted data access request for a token [3].

[0387] Next, the processor 310 receives the node values ​​of the input layer 1310 and the weight values ​​of the first connection network 1320 to calculate the node values ​​of the first hidden layer 1330. When the processor 310 generates a data access request for the token [3], the artificial neural network memory controller 320 determines whether the predicted data access request of the token [3] is the same as the data access request of the token [3]. When it is determined that the requests are the same, the calculated node values ​​of the first hidden layer 1330 can be stored in the memory 330 and / or the cache memory 322.

[0388] The cache memory 322 will be described separately. When the same data as a token [3] is stored in the memory 330 without the cache memory 322, and then the memory access request as a token [4] is read from the memory 330, the latency of the memory 330 can be doubled.

[0389] In this case, the artificial neural network memory controller 320 stores the node value of the layer calculated based on the fact that the memory address values ​​of consecutive tokens are the same and the operation mode of the previous token is write mode, the operation mode of the subsequent token is read mode, and determines to use the corresponding node value as the input value of the subsequent layer.

[0390] In other words, when the data of token [3] is stored in cache memory 322, data access requests corresponding to token [3] and token [4] can be processed in cache memory 322. Therefore, the artificial neural network memory controller 320 can be configured not to generate memory access requests corresponding to the data access requests of token [3] and token [4]. According to the above configuration, the memory-to-memory latency of token 330 can be eliminated through the memory access requests of token [3] and token [4]. In particular, the cache memory 322 operation strategy can be executed based on the artificial neural network data locality mode 1400.

[0391] At this time, the artificial neural network memory controller 320 generates a token[4] predicting the data access request.

[0392] Next, when the processor 310 generates a data access request for the token [4], the artificial neural network memory controller 320 determines whether the predicted data access request for the token [4] is the same as the data access request for the token [4]. When it is determined that the requests are the same, the node value of the first hidden layer 1330 stored in the cache memory 322 can be immediately provided to the processor 310.

[0393] At this time, the artificial neural network memory controller 320 generates a token[5] predicting the data access request.

[0394] Accordingly, the artificial neural network memory controller 320 generates a memory access request for a token [5] to store the weight values ​​of the second connection network 1340 in advance in the cache memory 322.

[0395] Next, when the processor 310 generates a data access request for the token [5], the artificial neural network memory controller 320 determines whether the predicted data access request for the token [5] is the same as the data access request for the token [5]. When it is determined that the requests are the same, the weight values ​​of the second connection network 1340 stored in the cache memory 322 can be immediately provided to the processor 310.

[0396] At this time, the artificial neural network memory controller 320 generates a predicted data access request for a token [6].

[0397] Next, processor 310 receives the node values ​​of the first hidden layer 1330 and the weight values ​​of the second connection network 1340 to calculate the node values ​​of the second hidden layer 1350. When processor 310 generates a data access request for token [6], artificial neural network memory controller 320 determines whether the predicted data access request for token [6] is the same as the data access request for token [6]. When the requests are determined to be the same, the calculated node values ​​of the second hidden layer 1350 can be stored in memory 330 and / or cache memory 322.

[0398] At this time, the artificial neural network memory controller 320 generates a predicted data access request for a token [7].

[0399] Next, when the processor 310 generates a data access request for the token [7], the artificial neural network memory controller 320 determines whether the predicted data access request for the token [7] is the same as the data access request for the token [7]. When it is determined that the requests are the same, the node value of the second hidden layer 1350 stored in the cache memory 322 can be immediately provided to the processor 310.

[0400] At this time, the artificial neural network memory controller 320 generates a predicted data access request for a token [8].

[0401] Accordingly, the artificial neural network memory controller 320 generates a memory access request for a token [8] to pre-store the weight values ​​of the third connection network 1360 in the cache memory 322.

[0402] Next, when the processor 310 generates a data access request for the token [8], the artificial neural network memory controller 320 determines whether the predicted data access request for the token [8] is the same as the data access request for the token [8]. When it is determined that the requests are the same, the weight values ​​of the third connection network 1360 stored in the cache memory 322 can be immediately provided to the processor 310.

[0403] At this time, the artificial neural network memory controller 320 generates a predicted data access request for a token [9].

[0404] Next, processor 310 receives the node values ​​of the second hidden layer 1350 and the weight values ​​of the third connection network 1360 to calculate the node values ​​of the output layer 1370. When processor 310 generates a data access request for token [9], artificial neural network memory controller 320 determines whether the predicted data access request for token [9] is the same as the data access request for token [9]. When the requests are determined to be the same, the calculated node values ​​of output layer 1370 can be stored in memory 330 and / or cache memory 322.

[0405] Therefore, the artificial neural network memory system 300 can store the inference results of the artificial neural network model 1300 in the output layer 1370.

[0406] Even if the inference of the artificial neural network model 1300 ends with the artificial neural network data locality pattern 1400, the artificial neural network memory system 300 can be ready to start the next inference immediately.

[0407] In other words, Figure 11 The artificial neural network memory system 300 can be configured to generate predicted data access requests based on the data locality of the artificial neural network, determine whether the predicted data access request is the same as the actual data access request, and if the requests are the same, further generate the next predicted data access request. According to the above configuration, the artificial neural network memory controller 320 can eliminate or reduce the latency of the memory 330 when processing data access requests.

[0408] In some examples, the artificial neural network memory controller can be configured to operate by generating at least one predicted data access request to minimize the available space of the cache memory.

[0409] In other words, the artificial neural network memory controller compares the available memory space of the cache with the size of the data value to be stored, and when the available memory space of the cache exists, generates at least one predicted data access request to minimize the available cache space.

[0410] In other words, an artificial neural network memory controller can be configured to generate multiple predicted data access requests based on the capacity of the cache memory.

[0411] In other words, an artificial neural network memory controller can be configured to sequentially generate at least one memory access request based on the remaining capacity of the cache memory to minimize the remaining capacity of the cache memory.

[0412] Reference Figures 2 to 6The example is described below. When the processor generates a data access request for token [1], the artificial neural network memory controller generates a predicted data access request for token [2] to pre-store the weight values ​​of the first connection network 1320 in the cache. Next, the artificial neural network memory controller may pre-allocate space for storing and retrieving the node value calculation results of the first hidden layer 1330 corresponding to tokens [3] and [4] in the cache. Next, the artificial neural network memory controller may pre-store the weight values ​​of the second connection network 1340 corresponding to token [5] in the cache. When there is spare capacity in the cache, the artificial neural network memory controller may be configured to further generate predicted data access requests sequentially according to the artificial neural network data locality pattern. That is, when there is spare capacity in the cache, the artificial neural network memory controller may be configured to pre-store the weight values ​​in the cache according to the artificial neural network data locality pattern or pre-secure the area for storing the results of artificial neural network operations.

[0413] If the cache memory has sufficient capacity, the weight values ​​of all connections in the artificial neural network model 1300 can be stored in the cache memory. Specifically, in the case of a fully learned artificial neural network model, the weight values ​​are fixed. Therefore, when the weight values ​​reside in the cache memory, the memory latency caused by reading the weight values ​​through memory access requests can be eliminated.

[0414] Based on the above configuration, the data required by the artificial neural network data locality storage cache is used to optimize the operating efficiency of the cache and improve the processing speed of the artificial neural network memory system 300.

[0415] Based on the above configuration, considering both the locality pattern of artificial neural network data and the capacity of the cache memory, the cache memory sequentially generates predicted data access requests, thereby improving the processing speed of the artificial neural network memory system.

[0416] According to the above configuration, when the processor generates a specific data access request included in the artificial neural network data locality pattern 1400, the artificial neural network memory controller can predict at least one data access request sequentially after the specific data access request. For example, when the processor generates a data access request for token [1], the artificial neural network memory controller can predict that the corresponding data access request is generated in the order of tokens [2-3-4-5-6-7-8-9].

[0417] According to the above configuration, the artificial neural network memory controller 320 can allow specific weight values ​​to reside in the cache memory for a specific period of time. For example, when the processor performs inference at a rate of 30 times per second using the artificial neural network model, the weight values ​​of a specific layer can reside in the cache memory. In this case, the artificial neural network memory controller can reuse the weight values ​​stored in the cache memory for each inference. Therefore, corresponding memory access requests can be selectively deleted. Thus, latency based on memory access requests can be eliminated.

[0418] In some examples, the cache memory may be configured with multiple hierarchical cache memories. For example, the cache memory may include a cache memory configured to store weight values ​​or a cache memory configured to store feature maps.

[0419] In some examples, when generating the artificial neural network data locality pattern 1400, the artificial neural network memory controller can be configured to predict weight values ​​and node values ​​based on identification information included in the data access request. Therefore, the artificial neural network memory controller can be configured to identify data access requests corresponding to weight values. Specifically, in the artificial neural network data locality pattern 1400, when it is assumed that learning is complete and the weight values ​​of the connected network are fixed, the weight values ​​can be configured to operate only in read mode. Therefore, the artificial neural network memory controller can identify tokens [2], [5], and [8] as weight values. In other words, token [1] is the starting step of inference, making it possible to identify it as an input node value. In other words, token [9] is the last step of inference, and therefore can be identified as an output node value. In other words, tokens [3] and [4] have the same memory address value in the order of write and read modes, making tokens [3] and [4] possible to identify as node values ​​of the hidden layer. However, this can vary depending on the artificial neural network data locality of the artificial neural network model.

[0420] Artificial neural network memory controllers can be configured to analyze the locality patterns of artificial neural network data to determine whether data access requests are weight values, kernel window values, node values, activation graph values, etc. of the artificial neural network model.

[0421] In some examples, an artificial neural network memory system includes: a processor configured to generate data access requests corresponding to artificial neural network operations; an artificial neural network memory controller configured to store artificial neural network data locality patterns generated by a compiler and to generate predicted data access requests that predict subsequent data access requests generated by the processor based on the artificial neural network data locality patterns; and a memory configured to communicate with the artificial neural network memory controller. The memory can be configured to operate according to memory access requests output from the artificial neural network memory controller.

[0422] Based on the above configuration, the artificial neural network memory controller can be configured to have an artificial neural network data locality pattern generated from the compiler. In this case, the artificial neural network memory controller can allow data access requests from the artificial neural network model being processed by the processor to be prepared in advance in the cache memory based on the artificial neural network data locality pattern generated by the compiler. Specifically, the artificial neural network data locality pattern generated by the compiler may be more accurate than the artificial neural network data locality pattern generated by monitoring artificial neural network data locality.

[0423] In other words, the artificial neural network memory controller can be configured to store artificial neural network data locality patterns generated by the compiler and artificial neural network data locality patterns generated by independently monitoring data access requests, respectively.

[0424] Figure 12 The illustration shows exemplary identification information for a data access request.

[0425] A data access request generated by a processor according to an example of this disclosure can be configured to further include at least one piece of additional identification information. This additional identification information may also be referred to as a sideband signal or sideband information.

[0426] Data access requests generated by the processor can be interface signals with a specific structure. That is, data access requests can be interface signals used for communication between the processor and the artificial neural network memory controller. Data access requests can be configured to further include additional bits to provide identification information required for the operation of the artificial neural network, but this disclosure is not limited thereto, and the additional identification information can be provided in various ways.

[0427] In some examples, data access requests for artificial neural network memory systems can be configured to further include identification information to identify whether they are artificial neural network operations, but the examples in this disclosure are not limited thereto.

[0428] For example, an artificial neural network memory system adds a single bit of identification code to a data access request to identify whether the data access request received by the artificial neural network memory controller is a data access request related to the operation of an artificial neural network. However, the number of bits in the identification code according to the examples of this disclosure is not limited and can be adjusted according to the number of cases of objects to be identified.

[0429] For example, when the identification code is [0], the artificial neural network memory controller can determine that the corresponding data access request is related to the artificial neural network operation.

[0430] For example, when the identification code is [1], the artificial neural network memory controller can determine that the corresponding data access request is unrelated to the artificial neural network operation.

[0431] In this configuration, the artificial neural network memory controller can be configured to generate artificial neural network data locality patterns by recording only data access requests related to the operation of the artificial neural network based on identification information included in the data access requests. According to the above configuration, the artificial neural network memory controller may not record data access requests unrelated to the operation of the artificial neural network. Doing so can improve the accuracy of the artificial neural network data locality patterns generated by recording data access requests, but the examples in this disclosure are not limited thereto.

[0432] In some examples, data access requests for an artificial neural network memory system can be configured to further include identification information to identify whether an artificial neural network operation is for learning or for inference, but the examples in this disclosure are not limited thereto.

[0433] For example, an artificial neural network memory system adds a single bit of identification code to data access requests, such that data access requests received through the artificial neural network memory controller are configured to identify whether the operation type of the artificial neural network model is learning or inference. However, the number of bits in the identification code according to the examples of this disclosure is not limited and can be adjusted according to the number of cases to be identified.

[0434] For example, when the identification code is [0], the artificial neural network memory controller can determine that the corresponding data access request is a learning operation.

[0435] For example, when the identification code is [1], the artificial neural network memory controller can determine that the corresponding data access request is an inference operation.

[0436] In this scenario, the artificial neural network memory controller can be configured to generate artificial neural network data locality patterns by separately recording data access requests for learning operations and inference operations. For example, the learning pattern may also include updating the weight values ​​of each layer and / or kernel window of the artificial neural network model, and determining an evaluation step to assess the inference accuracy of the trained artificial neural network model. Therefore, even if the artificial neural network models have the same structure, the artificial neural network data locality to be processed by the processor during learning and inference operations may differ.

[0437] According to the above configuration, the artificial neural network memory controller can be configured to generate artificial neural network data locality patterns for the learning mode and inference mode of a specific artificial neural network model, respectively. By doing so, the accuracy of the artificial neural network data locality patterns generated by recording data access requests by the artificial neural network memory controller can be improved, but the examples in this disclosure are not limited thereto.

[0438] In some examples, data access requests for an artificial neural network memory system can be configured with an operating mode that includes identification information to identify memory read operations and memory write operations, but is not limited thereto, such that the data access requests for an artificial neural network memory system can be configured with an operating mode that also includes identification information for identifying overwrite operations and / or protection operations, but the examples in this disclosure are not limited thereto.

[0439] For example, a single bit of identification code is added to the data access request of an artificial neural network memory system to include both read and write operations. Alternatively, two bits of identification code are added to the data access request of an artificial intelligence network memory system to identify read, write, overwrite, and protection operations. However, the number of bits in the identification code according to the examples of this disclosure is not limited and can be adjusted according to the number of cases to be identified.

[0440] In other words, for the operation of an artificial neural network memory system, data access requests need to include identification information to identify memory address values ​​and read and write operations. The artificial neural network memory controller receives data access requests to generate corresponding memory access requests to execute memory operations.

[0441] For example, when the identification code is

[000] , the artificial neural network memory controller can be configured to identify the corresponding data access request as a read operation.

[0442] For example, when the identification code is

[001] , the artificial neural network memory controller can be configured to identify the corresponding data access request as a write operation.

[0443] For example, when the identification code is

[010] , the artificial neural network memory controller can be configured to identify the corresponding data access request as an overwrite operation.

[0444] For example, when the identification code is

[011] , the artificial neural network memory controller can be configured to identify the corresponding data access request as a protection operation.

[0445] For example, when the identification code is

[100] , the artificial neural network memory controller can be configured to identify the corresponding data access request as a read-burst operation.

[0446] For example, when the identification code is

[001] , the artificial neural network memory controller can be configured to identify the corresponding data access request as a write-burst operation. However, the examples in this disclosure are not limited thereto.

[0447] According to the above configuration, the artificial neural network memory controller controls the memory according to the read mode or write mode to provide the memory with various data of the artificial neural network model or to store the data in the memory.

[0448] Based on the above configuration, the artificial neural network memory controller can update the weight values ​​of specific layers during the learning operation of the artificial neural network through an overwrite operation. Specifically, the updated weight values ​​are stored in the same memory address, thus avoiding the allocation of new memory addresses. Therefore, the overwrite mode may be more efficient than the write mode during the learning operation.

[0449] Based on the above configuration, the artificial neural network memory controller can protect data stored at specific memory addresses through a protection mode. Specifically, in environments where multiple users access the system simultaneously (such as servers), the data of the artificial neural network model cannot be arbitrarily deleted. Furthermore, the protection mode can also protect the weight values ​​of an artificial neural network model that has completed its learning process.

[0450] In some examples, data access requests for an artificial neural network memory system can be configured to further include identification information, weights, feature maps, learning datasets, evaluation datasets, etc., which can identify inference data, but the examples disclosed herein are not limited thereto.

[0451] For example, an artificial neural network memory system can be configured to add a 3-bit identification code to a data access request so that the artificial neural network memory controller can identify the field of the data to be accessed. However, the number of bits in the identification code according to the examples of this disclosure is not limited and can be adjusted according to the number of objects to be identified.

[0452] For example, when the identification code is

[000] , the artificial neural network memory controller can determine that the corresponding data is data that is unrelated to the artificial neural network model.

[0453] For example, when the identification code is

[001] , the artificial neural network memory controller can determine that the corresponding data is the inference data of the artificial neural network model.

[0454] For example, when the identification code is

[010] , the artificial neural network memory controller can determine that the corresponding data is the feature map of the artificial neural network model.

[0455] For example, when the identification code is

[011] , the artificial neural network memory controller can determine that the corresponding data is the weight of the artificial neural network model.

[0456] For example, when the identification code is

[100] , the artificial neural network memory controller can determine that the corresponding data is the learning dataset of the artificial neural network model.

[0457] For example, when the identification code is

[101] , the artificial neural network memory controller can determine that the corresponding data is the inference dataset of the artificial neural network model.

[0458] According to the above configuration, the artificial neural network memory controller can be configured to identify the domains of the artificial neural network model's data and allocate addresses for memory corresponding to the domains. For example, the artificial neural network memory controller can set the start and end addresses of the memory regions allocated to the domains. According to the above configuration, the data allocated to the domains can be stored in an order corresponding to the locality patterns of the artificial neural network data.

[0459] For example, data for a domain of an artificial neural network model can be stored sequentially in a memory area allocated to that domain. In this case, the memory can be a memory that supports read burst functionality. According to the above configuration, when the artificial neural network memory controller reads data from a specific domain of the memory, that specific data can be configured to be stored according to the locality pattern of artificial neural network data to optimize for read burst functionality. That is, the artificial neural network memory controller can be configured to set the memory's storage area in consideration of read burst functionality.

[0460] In some examples, the memory also includes a read burst function, and at least one artificial neural network memory controller can be configured to write to a storage area of ​​at least one memory in consideration of the read burst function.

[0461] In some examples, data access requests for an artificial neural network memory system can be configured to further include identification information to identify the quantization of the artificial neural network model, but the examples in this disclosure are not limited thereto.

[0462] For example, when a data access request includes at least a memory address value, a domain, and quantization identification information, an artificial neural network memory system can be configured to identify the quantization information of the data in the domain.

[0463] For example, when the identification code is

[00001] , the artificial neural network memory controller can determine that the corresponding data is quantized to one bit.

[0464] For example, when the identification code is

[11111] , the artificial neural network memory controller can determine that the corresponding data is quantized to 32 bits.

[0465] In some examples, various identification information may be selectively included in the data access request.

[0466] Based on the above configuration, the artificial neural network memory controller analyzes the identification codes of data access requests to generate more accurate artificial neural network data locality patterns. Furthermore, it calculates each identification code to selectively control the memory's storage strategy.

[0467] For example, when recognizing learning and reasoning, each artificial neural network data locality pattern can be generated.

[0468] For example, when the domain of the data is identified, a strategy can be developed to store the locality patterns of artificial neural network data in a specific memory region to improve the efficiency of memory operations.

[0469] In some examples, when the artificial neural network memory system is configured to process multiple artificial neural network models, the artificial neural network memory controller can be configured to further generate identification information for the artificial neural network models, such as additional identification information, for example, for a first artificial neural network model or a second artificial neural network model. In this case, the artificial neural network memory controller can be configured to distinguish the artificial neural network models based on the artificial neural network data locality, but this disclosure is not limited thereto.

[0470] Figure 12 The sideband signals and locality information of the artificial neural network (ANN) data shown can be selectively integrated or separated.

[0471] Artificial neural network computation: can determine whether the corresponding ANN operation was performed in the SAM memory controller.

[0472] Operation type: The SAM memory controller can determine whether the corresponding data is for training or inference (the schedule for weight value updates in inference mode).

[0473] Operating modes: RAM can be controlled within the SAM memory controller (in kernel mode, it can see domain refresh, while in feature map mode, it can be read-discarded).

[0474] Domain: This can be the information required for the memory map settings in the SAM memory controller. (A domain can allocate the same data to a specific area based on ANN data locality information.)

[0475] Quantization: The SAM memory controller can provide quantization information for the corresponding data.

[0476] ANN Model #: The SAM memory controller can allocate each model to the memory map based on ANN data locality information. This ensures a minimum total data size for the ANN.

[0477] Multi-threaded: The SAM memory controller can share the kernel and allocate separate feature maps according to the number of threads in each ANN model.

[0478] ANN data locality: This refers to the information at a specific processing stage of ANN data locality information.

[0479] On the other hand, all sideband signals can be implemented as a PACKET.

[0480] Figure 13 It is a diagram used to illustrate the energy consumption of each unit operation in an artificial neural network memory system.

[0481] refer to Figure 13 The table schematically illustrates the energy consumption per unit operation of the artificial neural network memory system 300. Energy consumption can be interpreted as the energy allocated to memory access, addition operations, and multiplication operations.

[0482] "8b Add" refers to the 8-bit integer addition operation of the adder. The 8-bit integer addition operation may consume 0.03 pj of energy.

[0483] "16b Add" refers to the 16-bit integer addition operation of the adder. The 16-bit integer addition operation may consume 0.05 pj of energy.

[0484] "32b Add" refers to the 32-bit integer addition operation of the adder. The 32-bit integer addition operation may consume 0.1 pj of energy.

[0485] "16b FP Add" refers to the 16-bit floating-point addition operation of the adder. A 16-bit floating-point addition operation may consume 0.4 pj of energy.

[0486] "32b FP Add" refers to the 32-bit floating-point addition operation of the adder. A 32-bit floating-point addition operation may consume 0.9 pj of energy.

[0487] "8b Mult" refers to the multiplier's 8-bit integer multiplication operation. An 8-bit integer multiplication operation may consume 0.2 pj of energy.

[0488] "32b Mult" refers to the multiplier's 32-bit integer multiplication operation. A 32-bit integer multiplication operation may consume 3.1 pj of energy.

[0489] "16b FP Mult" refers to the multiplier's 16-bit floating-point multiplication operation. A 16-bit floating-point multiplication operation may consume 1.1 pj of energy.

[0490] "32b FP Mult" refers to the multiplier's 32-bit floating-point multiplication operation. A 32-bit floating-point multiplication operation may consume 3.7 pj of energy.

[0491] "32b SRAM Read" refers to a 32-bit data read access when the cache memory 322 of the artificial neural network memory system 300 is static random access memory (SRAM). Reading 32-bit data from cache memory 322 to processor 310 may consume 5pJ of power.

[0492] "32-bit DRAM Read" refers to a 32-bit data read access when the memory 330 of the artificial neural network memory system 300 is DRAM. Reading 32-bit data from memory 330 to processor 310 may consume 640 pJ of energy. The unit of energy is picojoule (pJ).

[0493] When comparing 32-bit floating-point multiplication and 8-bit integer multiplication performed by the artificial neural network memory system 300, the energy consumption per unit operation differs by approximately 18.5 times. When reading 32-bit data from memory 330 composed of DRAM and reading 32-bit data from cache memory 322 composed of SRAM, the energy consumption per unit operation differs by approximately 128 times.

[0494] In other words, from a power consumption perspective, the larger the data bit size, the greater the power consumption. Furthermore, power consumption increases more significantly when using floating-point operations than integer operations. Additionally, power consumption increases rapidly when reading data from DRAM.

[0495] In another example of the artificial neural network memory system 300 according to the present disclosure, the capacity of the cache memory 322 can be configured to be sufficient to store all the data values ​​of the artificial neural network model 1300.

[0496] The cache memory described in the example is not limited to SRAM. Examples of static memories capable of performing high-speed operations like SRAM include SRAM, MRAM, STT-MRAM, eMRAM, OST-MRAM, etc. Furthermore, MRAM, STT-MRAM, eMRAM, and OST-MRAM are static memories with non-volatile characteristics. Therefore, when the power supply to the artificial neural network memory system 300 is turned off and then restarted, the artificial neural network model 1300 does not need to be supplied from memory 330 again, but the examples according to this disclosure are not limited to this.

[0497] According to the above configuration, when the artificial neural network memory system 300 performs inference operations of the artificial neural network model 1300 based on the artificial neural network data locality pattern 1400, the power consumption caused by the read operation of the memory 330 will be significantly reduced.

[0498] Figure 14 This is a schematic diagram used to explain various examples of artificial neural network memory systems according to this disclosure.

[0499] In the following text, reference will be made to Figure 14 Describe various examples based on this disclosure. Figure 14 The number of cases that can be explained and executed according to various examples of this disclosure.

[0500] According to various examples of this disclosure, an artificial neural network memory system 400 includes at least one processor, at least one memory, and at least one artificial neural network memory controller (AMC). The at least one AMC is configured to include at least one processor and receive data access requests from the at least one processor to provide the memory access request to the at least one memory. The at least one AMC may be configured to be substantially identical to exemplary artificial neural network memory controllers 120, 220, and 320. However, it is not limited thereto, and one artificial neural network memory controller of the artificial neural network memory system 400 may be configured differently from other artificial neural network memory controllers. For ease of description, repeated descriptions of artificial neural network memory controllers 411, 412, 413, 414, 415, 416, and 517, as well as the aforementioned artificial neural network memory controllers 120, 220, and 320, will be omitted below.

[0501] At least one artificial neural network memory controller is configured to connect at least one processor and at least one memory. In this case, corresponding artificial neural network data locality may exist in the data transfer path between the at least one processor and the at least one memory. Therefore, the artificial neural network memory controller located in the data transfer path can be configured to extract the corresponding artificial neural network data locality patterns.

[0502] Each AMC can be configured to monitor each data access request to generate artificial neural network data locality patterns. The artificial neural network memory system 400 can be configured to include at least one processor. The at least one processor can be configured to process artificial neural network operations individually or in cooperation with other processors.

[0503] The artificial neural network memory system 400 can be configured to include at least one internal memory. The artificial neural network memory system 400 can be configured to be connected to at least one external memory. The internal or external memory may include dynamic RAM (DRAM), high-bandwidth memory (HBM), static RAM (SRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, ferroelectric RAM (FRAM), magnetic RAM (MRAM), hard disk, phase-change memory (phase-change RAM), etc., but this disclosure is not limited thereto.

[0504] The external memory (External MEM1, External MEM2) or internal memory (Internal MEM1, Internal MEM2) can communicate with the artificial neural network memory system 400 through the corresponding memory interface (External MEM I / F).

[0505] The processor (Processor 1) may include a bus interface unit (BIU) that communicates with the system bus.

[0506] The artificial neural network memory system 400 may include an external memory interface connected to an external memory (MEM). The external memory interface transmits memory access requests to at least one external memory of the artificial neural network memory system 400 and can receive data in response to a memory access request from at least one external memory. The configurations and functions disclosed in exemplary artificial neural network memory controllers 120, 220, and 320 are distributed to a plurality of artificial neural network memory controllers 411, 412, 413, 414, 415, 416, and 517 disposed at specific locations within the artificial neural network memory system 400. In some examples, a processor may be configured to include an artificial neural network memory controller.

[0507] In some examples, the memory may be DRAM, and in such cases, the artificial neural network memory controller may be configured to be included in the DRAM.

[0508] For example, at least one of the artificial neural network memory controllers 411, 412, 413, 414, 415, 416, and 517 may be configured to include a cache memory. Furthermore, the cache memory may be configured to be included in the processor, internal memory, and / or external memory.

[0509] For example, at least one of the artificial neural network memory controllers 411, 412, 413, 414, 415, 416 and 517 can be configured to be distributed in the data transfer path between the memory and the processor.

[0510] For example, an artificial neural network memory controller that can be implemented in an artificial neural network memory system 400 can be configured by one of the following: a separately configured artificial neural network memory controller 411; an artificial neural network memory controller 412 included in a system bus; an artificial neural network memory controller 413 configured as an interface to a processor; an artificial neural network memory controller 414 included in a package block between an internal memory interface and a system bus; an artificial neural network memory controller included in an internal memory interface; an artificial neural network memory controller 415 included in internal memory; an artificial neural network memory controller included in a memory interface corresponding to external memory; an artificial neural network memory controller 416 included in a package block between an external memory interface and a system bus; and / or an artificial neural network memory controller 517 included in external memory. However, the artificial neural network memory controllers according to the examples of this disclosure are not limited thereto.

[0511] For example, the locality patterns of individual artificial neural network data generated by the first artificial neural network memory controller 411 and the second artificial neural network memory controller 412 may be the same as or different from each other.

[0512] In other words, the first artificial neural network memory controller 411 can be configured to connect the first processor (Processor 1) and the first internal memory (internal MEM1) via a system bus. In this case, data locality of the first artificial neural network may exist in the data transmission path between the first processor (Processor 1) and the first internal memory (internal MEM1).

[0513] In this case, a third artificial neural network memory controller 413 is shown in the path. However, this is merely illustrative and the third artificial neural network memory controller 413 may be omitted. That is, when at least one artificial neural network memory controller is positioned between the processor and the memory, artificial neural network data locality patterns of the artificial neural network model processed by the processor can be generated.

[0514] In other words, the second artificial neural network memory controller 412 can be configured to connect the second processor (Processor 2) and the first external memory MEM1. In this case, data locality of the second artificial neural network may exist in the data transmission path between the second processor (Processor 2) and the first external memory MEM1.

[0515] For example, the first artificial neural network model processed by the first processor (Processor 1) could be an object recognition model, while the second artificial neural network model processed by the second processor (Processor 2) could be a speech recognition model. Therefore, the artificial neural network models may differ from each other, and the corresponding artificial neural network data locality patterns may also differ from each other.

[0516] In other words, the artificial neural network data locality patterns generated by the artificial neural network memory controllers 411, 412, 413, 414, 415, 416 and 517 can be determined based on the pattern characteristics of the data access requests generated by the corresponding processors.

[0517] In other words, even if the artificial neural network memory controller of the artificial neural network memory system 400 is positioned between any processor and any memory, the artificial neural network memory controller can still provide adaptability to generate artificial neural network data locality patterns at appropriate locations. In other words, when two processors collaborate to process an artificial neural network model in parallel, the artificial neural network data locality patterns of the artificial neural network model can be partitioned and assigned to each processor. For example, the convolution operation of the first layer is processed by the first processor, and the convolution operation of the second layer is processed by the second processor, to distribute the operations of the artificial neural network model.

[0518] In this scenario, even if the artificial neural network models are identical, the artificial neural network data locality of the models processed by different processors can be reconstructed within the units of data access requests. In this case, each artificial neural network memory controller can provide adaptability to generate artificial neural network data locality patterns corresponding to the data access requests of the processors processed by the artificial neural network memory controller.

[0519] Data access requests can be generated per word line. Artificial neural network (ANN) DL can be configured based on these data access requests.

[0520] Based on the above configuration, even if multiple artificial neural network memory controllers are distributed across multiple processors and multiple memories, the performance of the artificial neural network memory system 400 can be optimized by generating artificial neural network data locality patterns suitable for each situation. In other words, each artificial neural network memory controller analyzes artificial neural network data locality at its location to optimize for real-time variable processing of artificial neural network operations.

[0521] In some examples, at least one of the artificial neural network memory controllers 411, 412, 413, 414, 415, 416 and 517 can be configured to confirm at least one of the following information: memory quantity, memory type, effective memory bandwidth, memory latency, and memory size.

[0522] In some examples, at least one of the artificial neural network memory controllers 411, 412, 413, 414, 415, 416, and 517 can be configured to measure the effective bandwidth of the memory in response to a memory access request. Here, the memory can be at least one memory, and each artificial neural network memory controller can measure the effective bandwidth of the channel communicating with each memory. The effective bandwidth can be calculated by measuring the time when the artificial neural network memory controller generates a memory access request and the time when the memory access request ends, and the data transfer bit rate.

[0523] In some examples, at least one of the artificial neural network memory controllers 411, 412, 413, 414, 415, 416 and 517 can be configured to have the necessary bandwidth for at least one memory in response to a memory access request.

[0524] In some examples, the artificial neural network memory system 400 includes multiple memories, and at least one artificial neural network memory controller can be configured to measure the effective bandwidth of the multiple memories.

[0525] In some examples, the artificial neural network memory system 400 includes multiple memories and at least one artificial neural network memory controller can be configured to measure the latency of the multiple memories.

[0526] In other words, at least one artificial neural network memory controller can be configured to perform automatic calibration of the memories connected to it. Automatic calibration can be configured to be performed at startup of the artificial neural network memory system or at specific intervals. At least one artificial neural network memory controller can be configured to collect information such as the number of memories connected to it, the type of memory, the effective bandwidth of the memory, the latency of the memory, and the size of the memory through automatic calibration.

[0527] Based on the above configuration, the artificial neural network memory system 400 can know the latency and effective bandwidth of the memory corresponding to the artificial neural network memory controller.

[0528] According to the above configuration, even if a separate artificial neural network memory controller is connected to the system bus, the artificial neural network data locality of the artificial neural network model being processed by the processor is also generated to control the memory.

[0529] In some examples, at least one artificial neural network memory controller of the artificial neural network memory system 400 can be configured to calculate the time taken to repeat an artificial neural network data locality pattern once, as well as the computational data size, to calculate the effective bandwidth required for artificial neural network operation. Specifically, the processor determines that the inference of the artificial neural network model has been completed after all data access requests contained in the artificial neural network data locality pattern have been processed. The artificial neural network memory system 400 can be configured to measure the time taken to perform one inference based on the artificial neural network data locality pattern to calculate inferences per second (IPS). Furthermore, the artificial neural network memory system 400 can be provided with target inferences per second information from the processor. For example, a particular application requires 30 IPS as the inference rate for a particular artificial neural network model. If the measured IPS is lower than the target IPS, the artificial neural network memory controller 400 can be configured to operate to improve the processor's artificial neural network model processing speed.

[0530] In some examples, the artificial neural network memory system 400 may be configured to include a system bus configured to control communication between the artificial neural network memory controller, the processor, and the memory. Further, at least one artificial neural network memory controller may be configured to have master control of the system bus.

[0531] In other words, the artificial neural network memory system 400 may not be a dedicated device for operating artificial neural networks. In this case, the system bus of the artificial neural network memory system 400 can connect to various peripheral devices, such as Wi-Fi devices, displays, cameras, or microphones. In this scenario, the artificial neural network memory system 400 can be configured to control the bandwidth of the system bus to ensure stable operation of the artificial neural network.

[0532] In some examples, at least one artificial neural network memory controller can be operated to prioritize artificial neural network operations during the processing time of memory access requests and process operations other than artificial neural network operations during the remaining time.

[0533] In some examples, at least one artificial neural network memory controller can be configured to ensure effective bandwidth of the system bus until at least one memory completes a memory access request.

[0534] In some examples, at least one artificial neural network memory controller is located in the system bus, and the system bus can be configured to dynamically change its bandwidth based on the locality patterns of artificial neural network data generated in the system bus.

[0535] In some examples, at least one artificial neural network memory controller is located on the system bus, and at least one artificial neural network memory controller can be configured to increase the control authority of the system bus to a level higher than the control authority when there is no memory access request, until at least one memory completes a response to the memory access request.

[0536] In some examples, at least one artificial neural network memory controller can be configured to prioritize data access requests from processors that handle artificial neural network operations over data access requests from processors that handle operations other than artificial neural network operations.

[0537] In some examples, the artificial neural network memory controller can be configured to directly control the memory.

[0538] In some examples, an artificial neural network memory controller is included in the memory, and the artificial neural network memory controller can be configured to generate at least one access queue. The artificial neural network memory controller can also be configured to generate a separate access queue dedicated to artificial neural network operations.

[0539] In some examples, at least one of the multiple memories can be SAM or DRAM. In this case, at least one artificial neural network memory controller can be configured to reorder the access queues of memory access requests. Access queue reordering can be a reclassification of the access queues.

[0540] In some examples, the artificial neural network memory controller can be configured to include access queues for multiple memory access requests. In this case, the first access queue may be a dedicated access queue for artificial neural network operations, while the second access queue may be an access queue for operations other than artificial neural network operations. The artificial neural network memory controller can be configured to provide data by selecting each access queue according to a priority setting.

[0541] In some examples, at least one artificial neural network memory controller can be configured to process specific memory access requests based on artificial neural network data locality patterns using a specific bandwidth required by the computing system bus, and at least one artificial neural network memory controller can be configured to control the effective bandwidth of the system bus based on a specific bandwidth.

[0542] Based on the above configuration, the artificial neural network memory system 400 can be configured to reduce the priority of memory access requests from various peripheral devices or increase the priority of early data access requests based on the data locality pattern of artificial neural networks.

[0543] Based on the above configuration, the artificial neural network memory controller readjusts the processing order of data access requests on the system bus to fully utilize the bandwidth of the system bus while processing artificial neural network operations, and to generate bandwidth for processing data of other peripheral devices when there are no artificial neural network operations.

[0544] Based on the above configuration, the artificial neural network memory controller can readjust the processing order of data access requests based on the data locality pattern of the artificial neural network. Furthermore, the artificial neural network memory controller readjusts priorities based on the identification information contained in the data access requests. In other words, from the perspective of artificial neural network operation, the effective bandwidth of the system bus is dynamically changing, thereby increasing the effective bandwidth. Therefore, the operating efficiency of the system bus can be improved. Thus, from the perspective of the artificial neural network memory controller, the effective bandwidth of the system bus can be increased.

[0545] In some examples, at least one artificial neural network memory controller can be configured to perform machine learning on data access requests. That is, at least one artificial neural network memory controller may also include an artificial neural network model configured to machine learn patterns of artificial neural network data locality. In other words, the artificial neural network data locality patterns are machine learned to learn specific patterns (thus interrupting another data access request during data access request processing based on actual artificial neural network data locality) for prediction.

[0546] When a predicted data access request is generated, the artificial neural network model embedded in the artificial neural network memory controller can be machine trained to elevate the control authority of the system bus to a level higher than that when the predicted data access request is not generated.

[0547] In some examples, at least one artificial neural network memory controller also includes multiple hierarchical caches, and at least one artificial neural network memory controller can be configured to perform machine learning for data access requests between layers of the multiple hierarchical caches.

[0548] In some examples, at least one artificial neural network memory controller can be configured to have at least one of the effective bandwidth, power consumption, and latency information for each layer of a plurality of hierarchical cache memories.

[0549] Based on the above configuration, the artificial neural network memory controller can be configured to generate artificial neural network data locality patterns through machine learning. These machine-generated artificial neural network data locality patterns can improve the probability of predicting the occurrence of a specific pattern when various data access requests unrelated to artificial neural network operations are generated in that pattern. Furthermore, reinforcement learning is used to predict the characteristics of various artificial neural network models and other operations processed by the processor to improve the efficiency of artificial neural network operations.

[0550] In some examples, at least one artificial neural network memory controller can be configured to partition and store data to be stored in multiple memories based on the effective bandwidth and latency of each of the multiple memories.

[0551] For example, data is configured in L-bit groups, and multiple memories include a first memory and a second memory. The first memory is configured to partition and store M-bit data from the L-bit groups based on a first effective bandwidth or a first delay, and the second memory is configured to partition and store N-bit data from the L-bit groups based on a second effective bandwidth or a second delay. The sum of the M-bits and N-bits can be configured to be less than or equal to L bits. Further, the multiple memories also include a third memory, and the third memory is configured to store O-bit data from the L-bit groups based on a third effective bandwidth or a third delay, and the sum of the M-bits, N-bits, and O-bits can be configured to be equal to L bits.

[0552] For example, the data consists of P data packets, and multiple memories include a first memory and a second memory. The first memory is configured to store R data packets out of the P data packets based on a first effective bandwidth or a first delay, and the second memory is configured to store S data packets out of the P data packets based on a second effective bandwidth or a second delay. The sum of R and S can be configured to be less than or equal to P. Additionally, the multiple memories also include a third memory, and the third memory is configured to store T data packets from the P data packets based on a third effective bandwidth or a third delay, where the sum of R, S, and T can be configured to be equal to P.

[0553] Based on the above configuration, when the bandwidth of a memory is low, the artificial neural network memory controller can allocate the data to be stored or retrieved, thereby increasing the effective bandwidth of the memory. For example, the artificial neural network memory controller can be configured to divide the 8 bits of the quantized weight value into 4 bits stored or retrieved in a first memory and 4 bits stored or retrieved in a second memory. Therefore, from the perspective of the artificial neural network memory controller, the effective bandwidth of the memory can be improved.

[0554] The artificial neural network memory controller can be configured to further include a cache memory configured to merge and store data that has been partitioned and stored across multiple memories. That is, at least one artificial neural network memory controller also includes a cache memory and can be configured to merge data distributed across multiple memories to store the merged data in the cache memory. Therefore, the merged data can be provided to the processor.

[0555] To merge the partitioned data, at least one artificial neural network memory controller can be configured to store partitioning information of the data that has been partitioned into multiple memories. Various examples of this disclosure will be described below.

[0556] According to one example of this disclosure, an artificial neural network memory system can be configured to include: at least one processor configured to generate data access requests corresponding to artificial neural network operations; and at least one artificial neural network memory controller configured to generate an artificial neural network data locality pattern of artificial neural network operations by sequentially recording data access requests, and to generate predicted data access requests based on the artificial neural network data locality pattern, the predicted data access requests predicting subsequent data access requests of the data access requests generated by the at least one processor. Here, artificial neural network data locality is artificial neural network data locality reconstructed at the processor-memory level.

[0557] According to an example of this disclosure, an artificial neural network memory system can be configured to include: at least one processor configured to process an artificial neural network model; and at least one artificial neural network memory controller configured to store artificial neural network data locality information of the artificial neural network model, and to predict data to be requested by the at least one processor based on the artificial neural network data locality information to generate a predicted data access request.

[0558] An artificial neural network memory system may be configured to further include at least one memory and a system bus configured to control communication between an artificial neural network memory controller, at least one processor, and at least one memory. According to an example of this disclosure, the artificial neural network memory system includes a processor, a memory, and a cache memory, and is configured to generate a predicted data access request, including data to be requested by the processor, based on artificial neural network data locality information, and to store data corresponding to the predicted data access request from the memory in the cache memory prior to the processor request.

[0559] According to examples of this disclosure, an artificial neural network memory system can be configured to operate in either a first mode or a second mode, wherein the first mode is configured to operate by receiving artificial neural network data locality information, and the second mode is configured to operate by observing data access requests generated by a processor to predict artificial neural network data locality information.

[0560] At least one artificial neural network memory controller can be configured to generate predicted data access requests sequentially based on artificial neural network data locality patterns.

[0561] At least one artificial neural network memory controller can be configured to generate a predicted data access request before generating a subsequent data access request.

[0562] At least one processor can be configured to send a data access request to at least one artificial neural network memory controller.

[0563] At least one artificial neural network memory controller can be configured to output a predicted data access request in response to a data access request.

[0564] Data access requests can be configured to further include memory addresses.

[0565] Data access requests can be configured to further include the start and end addresses of the memory.

[0566] At least one artificial neural network memory controller can be configured to generate a memory access request based on either a data access request generated by at least one processor or a predicted data access request generated by the artificial neural network memory controller.

[0567] The data access request can be configured to further include the starting address of the memory and a sequential data read trigger.

[0568] Data access requests can be configured to further include the starting address of the memory and the amount of contiguous data.

[0569] Data access requests and predicted data access requests can be configured to also include a data access request token with the same matching memory address.

[0570] Data access requests can be configured to further include identification information to determine whether they are memory read or write commands.

[0571] Data access requests can be configured to further include identification information to determine whether they are memory overwrite commands.

[0572] Data access requests can be configured to further include identification information to determine whether they are inference data, weighted data, or feature map data.

[0573] Data access requests can be configured to further include identification information to determine whether the data is for learning or evaluation purposes.

[0574] Data access requests can be configured to further include identification information to determine whether an artificial neural network operation is for learning or for inference.

[0575] When at least one processor generates a subsequent data access request, at least one artificial neural network memory controller can be configured to determine whether the predicted data access request and the subsequent data access request are the same request.

[0576] When the predicted data access request and the subsequent data access request are the same request, at least one artificial neural network memory controller can be configured to maintain the data locality pattern of the artificial neural network.

[0577] When the predicted data access request differs from the subsequent data access request, at least one artificial neural network memory controller can be configured to update the artificial neural network data locality pattern.

[0578] The data locality pattern of an artificial neural network can be configured to further include data in which the addresses of memory access requests are sequentially recorded.

[0579] At least one artificial neural network memory controller can be configured to generate artificial neural network data locality patterns by detecting recurring patterns of memory addresses included in data access requests.

[0580] The data locality pattern of artificial neural networks can be configured by memory addresses with repetitive loop characteristics.

[0581] The data locality pattern of artificial neural networks can be configured to further include identification information for recognizing the start and end of operations in the artificial neural network model.

[0582] At least one processor can be configured to be provided with data corresponding to a data access request from an artificial neural network memory controller.

[0583] At least one artificial neural network memory controller can be configured to further include an artificial neural network model configured to machine learning artificial neural network data locality patterns.

[0584] At least one artificial neural network memory controller can be configured to store update modes and advance modes of locality patterns of artificial neural network data to determine whether the artificial neural network model has changed.

[0585] At least one artificial neural network memory controller can be configured to determine whether a data access request is a request from one artificial neural network model or a mixture of requests from multiple artificial neural network models.

[0586] When multiple artificial neural network models exist, at least one artificial neural network memory controller can be configured to further generate artificial neural network data locality patterns corresponding to the number of artificial neural network models.

[0587] At least one artificial neural network memory controller can be configured to generate corresponding predicted data access requests individually based on the data locality pattern of the artificial neural network.

[0588] At least one artificial neural network memory controller can be configured to further generate data access requests corresponding to data access requests.

[0589] At least one artificial neural network memory controller can be configured to further generate data access requests corresponding to the predicted data access requests.

[0590] Each of the data access request, the predicted data access request, and the memory access request can be configured to include the corresponding memory address value and operating mode.

[0591] At least one artificial neural network memory controller can be configured to further generate a memory access request, which includes at least a portion of the information included in the data access request and the predicted data access request.

[0592] It also includes at least one memory configured to communicate with at least one artificial neural network memory controller, and the at least one memory can be configured to operate in response to a memory access request output from at least one artificial neural network memory controller.

[0593] At least one memory may be configured to store at least one of inference data, weight data, and feature map data.

[0594] At least one neural network artificial neural network memory controller may be configured to further include a cache memory configured to store data sent from at least one memory in response to a memory access request.

[0595] When at least one processor outputs a subsequent data access request, at least one artificial neural network memory controller determines whether the predicted data access request and the subsequent (i.e., the next) data access request are the same. If the predicted data access request and the subsequent data access request are the same, the at least one artificial neural network memory controller can be configured to provide data stored in a cache memory to at least one processor, and if the predicted data access request and the subsequent data access request are different, the at least one artificial neural network memory controller can be configured to generate a new memory access request based on the subsequent data access request.

[0596] At least one artificial neural network memory controller sequentially generates at least one memory access request based on the remaining capacity of the cache memory to minimize the remaining capacity of the cache memory.

[0597] At least one artificial neural network memory controller can be configured to measure the effective bandwidth of at least one memory in response to a memory access request.

[0598] At least one artificial neural network memory controller can be configured to have the necessary bandwidth for at least one memory in response to a memory access request.

[0599] At least one artificial neural network memory controller can be configured to measure the number of inferences per second (IPS) of an artificial neural network operation by calculating the number of repetitions of the locality pattern of artificial neural network data within a specific time period.

[0600] At least one artificial neural network memory controller can be configured to calculate the time taken to repeat the locality pattern of the artificial neural network data once and to calculate the data size in order to calculate the effective bandwidth required for the operation of the artificial neural network.

[0601] At least one memory also includes DRAM, which includes a refresh function to update the voltage of memory elements, and at least one artificial neural network memory controller can be configured to selectively control the refresh of the memory address region of at least one memory corresponding to a predicted data access request.

[0602] At least one memory also includes a pre-charge function to charge the global bit lines of the memory using a specific voltage, and at least one artificial neural network memory controller can be configured to selectively provide pre-charge to the memory address regions of at least one memory corresponding to a predicted data access request.

[0603] At least one memory also includes multiple memories, and at least one artificial neural network memory controller can be configured to measure the effective bandwidth of the multiple memories respectively.

[0604] At least one memory also includes multiple memories, and at least one artificial neural network memory controller can be configured to measure the latency of the multiple memories respectively.

[0605] At least one memory also includes multiple memories, and at least one artificial neural network memory controller can be configured to partition and store data to be stored in the multiple memories based on the effective bandwidth and latency of each of the multiple memories.

[0606] The data is configured in L-bit groups, and the multiple memories include a first memory and a second memory. The first memory is configured to divide and store M-bit data from the L-bit groups based on a first effective bandwidth or a first delay, and the second memory is configured to divide and store N-bit data from the L-bit groups based on a second effective bandwidth or a second delay. The sum of the M-bits and N-bits can be configured to be less than or equal to the L-bits.

[0607] The plurality of memories also includes a third memory, and the third memory is configured to store 0-bit data from a bit group of L bits based on a third effective bandwidth or a third delay, and the sum of M bits, N bits and 0 bits can be configured to be equal to L bits.

[0608] At least one artificial neural network memory controller may be configured to further include a cache memory configured to merge and store data that is partitioned to be stored in multiple memories.

[0609] The data consists of P data packets, and multiple memories include a first memory and a second memory. The first memory is configured to store R data packets out of the P data packets based on a first effective bandwidth or a first delay, and the second memory is configured to store S data packets out of the P data packets based on a second effective bandwidth or a second delay. The sum of R and S can be configured to be less than or equal to P.

[0610] The multiple memories also include a third memory, which is configured to store T data packets from P data packets based on a third effective bandwidth or a third delay, wherein the sum of R, S and T can be configured to equal P.

[0611] At least one memory also includes multiple memories, and at least one artificial neural network memory controller also includes a cache memory and is configured to merge data that is distributed and stored in multiple memories to store the merged data in the cache memory.

[0612] At least one memory also includes multiple memories, and at least one artificial neural network memory controller can be configured to store partitioning information of data that is divided to be stored in the multiple memories.

[0613] At least one artificial neural network memory controller can be configured to store a portion of the data in a cache memory as much as the latency, based on predicted data access requests and the latency value of at least one memory.

[0614] At least one artificial neural network memory controller can be configured to store a portion of the data in a cache memory based on predicted data access requests and required data bandwidth from at least one memory.

[0615] When at least one processor generates a subsequent data access request, at least one artificial neural network memory controller first provides the data stored in the cache memory and controls the remaining data from at least one memory in a read burst mode to reduce the latency of at least one memory.

[0616] When at least one processor generates subsequent data access requests based on predicted data access requests and latency values ​​of at least one memory, at least one artificial neural network memory controller starts in advance in a read burst mode of at least one memory as much as the latency value to reduce the latency of at least one memory.

[0617] It may further include a system bus configured to control communication between an artificial neural network memory controller, at least one processor, and at least one memory.

[0618] At least one artificial neural network memory controller can be configured to have master control of the system bus.

[0619] At least one artificial neural network memory controller also includes an artificial neural network model, and when a predicted data access request is generated, the artificial neural network model can be machine trained to increase the control of the system bus to a level higher than the control when no predicted data access request is generated.

[0620] At least one artificial neural network memory controller can be configured to ensure effective bandwidth of the system bus until at least one memory completes a memory access request.

[0621] At least one artificial neural network memory controller can be configured to process a specific memory access request using a specific bandwidth required by the artificial neural network data locality pattern computing system bus, and at least one artificial neural network memory controller can be configured to control the effective bandwidth of the system bus according to the specific bandwidth.

[0622] At least one artificial neural network memory controller is disposed in the system bus, and the system bus is configured to dynamically change the bandwidth of the system bus based on the locality patterns of artificial neural network data generated in the system bus.

[0623] At least one artificial neural network memory controller can be operated to prioritize artificial neural network operations during the processing time of memory access requests, and to process operations other than artificial neural network operations at other times.

[0624] At least one artificial neural network memory controller and at least one processor can be configured to communicate directly with each other.

[0625] The artificial neural network memory controller can be configured to further include a first access queue and a second access queue, the first access queue being an access queue dedicated to artificial neural network operations, and the second access queue being an access queue for other purposes besides artificial neural network operations, and the artificial neural network memory controller can be configured to select the access queue to provide data based on a priority setting.

[0626] At least one artificial neural network memory controller also includes multiple hierarchical caches, and at least one artificial neural network memory controller can be configured to further include an artificial neural network model configured to perform machine learning for data access requests between layers of the multiple hierarchical caches.

[0627] At least one artificial neural network memory controller can be configured to be further provided with at least one of the effective bandwidth, power consumption, and latency information for each layer of a plurality of hierarchical cache memories.

[0628] The system includes: at least one processor configured to generate data access requests corresponding to artificial neural network operations; at least one artificial neural network memory controller configured to store artificial neural network data locality patterns of artificial neural network operations generated from a compiler and to generate predicted data access requests, which predict subsequent data access requests based on the artificial neural network data locality patterns to predict data access requests generated by the at least one processor; and at least one memory configured to communicate with the at least one artificial neural network memory controller. The at least one memory may be configured to operate according to memory access requests output from the at least one artificial neural network memory controller.

[0629] At least one artificial neural network memory system may be configured to further include at least one memory and a system bus, the system bus being configured to control communication between the artificial neural network memory controller, at least one processor, and at least one memory.

[0630] At least one artificial neural network memory controller is disposed in the system bus, and at least one artificial neural network memory controller can be configured to elevate the control authority of the system bus to a level higher than the control authority when there is no memory access request, until at least one memory completes a response to the memory access request.

[0631] At least one artificial neural network memory controller includes one or more artificial neural network memory controllers configured to be included in DRAM.

[0632] At least one artificial neural network memory controller includes one or more artificial neural network memory controllers configured to be included in at least one processor.

[0633] At least one memory further includes DRAM, or at least one memory is DRAM, and at least one artificial neural network memory controller can be configured to reorder the access queue of memory access requests. That is, at least one artificial neural network memory controller can be configured to control the reordering threads of the DRAM memory controller.

[0634] Memory access requests related to artificial neural network operations provided from the artificial neural network memory controller to the memory controller of the memory may also include priority information that can be interpreted by the memory controller of the memory.

[0635] Based on the above configuration, the memory controller can be configured to reorder the memory access queues within the memory controller based on priority information included in the memory access requests generated by the artificial neural network memory controller, regardless of whether the memory access requests are related to artificial neural network operations. Therefore, the access queue for memory access requests used to process artificial neural network operations can be processed earlier than the access queue for other types of memory access requests. Consequently, the artificial neural network memory controller can increase the effective bandwidth of the corresponding memory.

[0636] The memory access request processing order determined by the DRAM memory controller can be readjusted using priority information provided by an artificial neural network memory controller.

[0637] For example, when the priority of a memory access request generated by an artificial neural network memory controller is set to urgent, the DRAM memory controller can change the processing sequence of the memory access request to the first priority.

[0638] The artificial neural network memory controller can be configured to generate at least one access queue.

[0639] At least one memory includes an artificial neural network memory controller, and the artificial neural network memory controller can be configured to generate a separate access queue dedicated to artificial neural network operations.

[0640] At least one artificial neural network memory controller can be configured to reorder the access queue of memory access requests.

[0641] At least one memory also includes a read burst function, and at least one artificial neural network memory controller can be configured to set up the storage area of ​​at least one memory in consideration of the read burst function.

[0642] At least one memory also includes a read burst function, and at least one artificial neural network memory controller can be configured to handle write operations in the storage area of ​​at least one memory in consideration of the read burst function.

[0643] The processor further includes a plurality of processors, and at least one artificial neural network memory controller can be configured to prioritize data access requests from processors that process artificial neural network operations over data access requests from processors that process operations other than artificial neural network operations.

[0644] For example, a processor according to this disclosure may be configured with one of the exemplary NPUs of this disclosure. For example, a SoC according to this disclosure may include an artificial neural network memory system. NPUs and SoCs will be described later.

[0645] At least one AMC can be configured to operate independently of each other based on ANN DL information stored separately in at least one AMC. The ANN DL information of each AMC may be the same or different from each other depending on its location in the system. The ANN DL information corresponding to each AMC can be configured to have at least partially the same information depending on its location in the system. Specifically, each AMC is configured to customize the ANN DL information of the ANN model being processed via a specific communication bus controlled by that specific AMC. In other words, the first ANN DL information of the first AMC on the first bus may differ from the second ANN DL information of the second AMC on the second bus. Therefore, each AMC has the advantage of being able to operate independently for a specific communication bus based on its ANN DL information.

[0646] Figure 15 This is an example diagram showing a substrate and channels on which a memory is mounted.

[0647] As shown in the figure, multiple pins for communicating with the memory bus can be formed on the substrate on which the memory is mounted, i.e., on the circuit board.

[0648] The memory bus may include an address bus (e.g., 17-bit), a command and control bus (e.g., 6-bit), and a data bus (e.g., 64-bit). More specifically, the memory bus may also include... Figure 12 At least one sideband signal is shown.

[0649] In other words, based on the added sideband signals, the SAM controller according to various examples of this disclosure can be configured to selectively classify and control memory element regions of the memory. However, this disclosure is not limited thereto, and IP header packets instead of sideband signals can be implemented.

[0650] This disclosure may provide a memory configured to operate according to the data locality (DL) principle of artificial neural networks (ANNs), such as a sequential access memory (SAM) and a SAM controller. The SAM may be referred to as a memory dedicated to artificial neural networks. The SAM control unit may refer to a memory controller that controls the SAM.

[0651] In other words, the SAM according to the examples of this disclosure can refer to a memory specifically designed for artificial neural network processing (which can exclude the random access characteristics of DRAM memory) and is configured to operate sequentially according to ANN DL information. However, the structure of the memory element of the SAM is not limited to DRAM; it can be applied to memories with memory element structures similar to DRAM. That is, based on ANN DL information, sequential access information that enables sequential access to memory addresses can be derived.

[0652] SAM can be configured to handle read / write commands primarily in burst mode. In this case, read / write commands can be configured to operate within a cell of ANN DL information. That is, the SAM controller can be configured to request memory operations from the SAM within a cell of ANN DL. In this case, the memory address mapping of SAM can be configured such that memory operations within a cell of ANN DL operate in basic burst mode without a specific burst mode command. Here, a cell of ANN DL can represent the smallest unit of data access request from the processor to memory or AMC based on ANN DL information. The minimum size of a cell of ANN DL can be a word unit, such as 16-bit, 32-bit, 64-bit, etc.

[0653] By providing ANN DL, SAM can essentially eliminate the random access characteristics of memory. Since SAM operates in a basic burst mode based on ANNDL information, the frequency of CAS latency and RAS latency can be minimized.

[0654] In other words, regular random access operations to memory are only effective when the sequence of memory operations from the processor is unpredictable.

[0655] On the other hand, SAM can know the sequence of memory operation requests requested by the processor in advance based on ANN DL. Therefore, SAM can provide memory operations with minimal power consumption and latency based on ANN DL.

[0656] The memory bus between the SAM and the SAM controller may further include at least one sideband signal.

[0657] The system bus between the SAM controller and the processor may further include at least one sideband signal. The number of sideband signals for the memory bus and the system bus may be the same or different from each other.

[0658] However, this disclosure is not limited thereto, and can be implemented in the form of a data packet that includes information corresponding to the sideband signal.

[0659] Because SAM excludes the random access characteristics of memory and operates based on ANN DL information, it can be configured to achieve precise refresh timing control of the memory elements of SAM. Dynamic memory elements may require periodic refresh, and dynamic memory implemented with SAM can be configured to control refresh based on ANN DL.

[0660] Because SAM excludes the random access characteristics of memory and operates based on ANN DL information, it can be configured to perform precise precharge timing control on the memory elements of SAM. Dynamic memory elements may require precharge for sense amplifier operation, and dynamic memory implemented with SAM can be configured to control precharge based on ANN DL.

[0661] SAM can be configured to determine the allocated memory region for each ANN DL message or for each domain.

[0662] Figure 16 This is an example diagram illustrating the process of reading data from a memory with a multi-memory structure.

[0663] Figure 16 The SAM shown can operate using some of the memory elements of a conventional DRAM. The SAM may include at least one memory bank.

[0664] refer to Figure 16 SAM can be configured such that the matrix-form memory elements have row and column addresses. The memory bank of SAM can be configured by bundling multiple memory elements.

[0665] To improve the bandwidth of SAM, SAM can be configured to interleave memory elements in each memory bank of SAM.

[0666] To increase the bandwidth of SAM, SAM can be configured to perform interleaving using the cells of the SAM's memory bank.

[0667] The row address strobe (RAS) and / or column address strobe (CAS) signals of the SAM's memory elements can be directly controlled based on the ANN DL information. Therefore, the SAM controller can control the SAM to read or write data according to the ANN DL sequence.

[0668] refer to Figure 12 , 15 Alternatively, 16, the SAM according to the examples of this disclosure may include multiple memory banks. In this case, the SAM can be configured to allocate specific regions of specific memory banks and / or memory elements for a specific purpose based on at least one sideband signal.

[0669] For example, depending on the domain, the first storage of SAM can be specifically allocated to the feature map.

[0670] For example, depending on the domain, the second storage bank of SAM may be specifically allocated to the kernel.

[0671] The SAM according to the examples of this disclosure may include at least one memory bank. In this case, the SAM can be configured to allocate a specific row of at least one memory bank for a specific purpose based on at least one sideband signal.

[0672] The SAM according to the examples of this disclosure may include at least one storage bank. In this case, the SAM may be configured to allocate a specific row of at least one storage bank for a specific purpose based on a domain.

[0673] For example, depending on the domain, the second storage bank of SAM can be specifically allocated to the kernel.

[0674] The SAM according to the examples of this disclosure may include at least one memory bank. In this case, the SAM can be configured to allocate a specific row of at least one memory bank for a specific purpose based on at least one sideband signal.

[0675] For example, SAM can assign rows of the first region of the first memory bank to the feature map based on the domain.

[0676] For example, SAM can assign weights to rows of the second region of the first storage based on the domain.

[0677] Refer again Figure 12 Alternatively, 15, the SAM according to the examples of this disclosure can be configured to allocate a specific row of a specific memory bank for a specific purpose based on the ANN MODEL# signal. The SAM according to the examples of this disclosure can be configured to allocate a specific row of a specific memory bank for a specific purpose based at least on the ANNDL signal. That is, the SAM can be configured to allocate memory elements of a specific memory bank or a specific row for a specific purpose based on at least one sideband signal.

[0678] However, this disclosure is not limited thereto, and even without additional sideband signals, the SAM controller can be implemented by directly controlling the memory address of the SAM based on ANN DL information.

[0679] Figure 17This is an example diagram illustrating the delay that occurs in conventional DRAM.

[0680] refer to Figure 17 This shows the latency between the CPU, the conventional memory controller, and the conventional DRAM.

[0681] Traditional CPUs use virtual memory, which utilizes a translation-backup buffer (TLB) to handle various operations. Therefore, artificial neural network data stored in conventional DRAM is segmented and stored in DRAM.

[0682] The CPU's operation of reading data from DRAM can include steps A through F. Each process introduces a delay.

[0683] In step A, the CPU generates a transaction request. During this process, the transaction request may be temporarily queued in the CPU, potentially causing delays. In step B, the CPU can transmit the transaction request to the memory controller. In step C, the memory controller can convert the transaction request into a sequence of instructions. In step D, the memory controller can transmit the sequence of instructions to the DRAM. In step E, the DRAM can process the sequence of instructions using a single CAS signal, a combination of RAS and CAS signals, or a combination of precharge (PRE), RAS, and CAS signals. In process F, the transaction data is transmitted to the CPU.

[0684] The delay from process A to process F may include A+B+C+D+E+F.

[0685] When all data corresponding to the data operation requested by traditional DRAM is latched Figure 31A When the sensing amplifier shown is in operation, process E1 will occur.

[0686] Process E2 occurs when a portion of the data corresponding to a data operation requested by conventional DRAM is segmented in a multi-row memory element.

[0687] Process E3 occurs when a portion of the data corresponding to a data operation requested by conventional DRAM is segmented into multiple rows of memory elements, and the memory elements are precharged for various reasons.

[0688] Here, RAS refers to the Row Address Strobe (RAS) signal, CAS refers to the Column Address Strobe (CAS) signal, and PRE refers to the Precharge signal. Each signal includes a corresponding delay.

[0689] When conventional DRAM and conventional memory controllers process artificial neural network (ANN) data, ANN deep learning (DL) information is not considered. Therefore, the ANN data is segmented and processed as virtual memory. Consequently, E2 and E3 errors occur more frequently than E1 errors in the conventional approach. Therefore, using conventional DRAM may become a bottleneck for ANN processing.

[0690] In contrast, in the case of SAM according to the example of this disclosure, since the operation is based on ANN DL, the occurrence frequency of E1 can be maximized by removing or minimizing the occurrence frequencies of E2 and E3. Therefore, the delay based on the CAS signal, RAS signal, and PRE signal can be reduced. Thus, the processing speed of the artificial neural network can be improved.

[0691] Figure 18 This is an example diagram illustrating the basic concept of Sequential Access Memory (SAM) according to this disclosure.

[0692] Figure 18 The main memory, SAM, SAM controller, and processor are shown. The SAM controller is located between the processor and the SAM to control the SAM.

[0693] The SAM controller can be integrated with the SAM, which serves as main memory, or it can be implemented physically separate from the SAM. Alternatively, the SAM controller can be embedded in the processor. Furthermore, the SAM controller can be implemented in various forms.

[0694] The SAM controller can receive ANN DL information from the processor to process artificial neural networks (ANNs), such as an NPU or compiler.

[0695] ANN DL information can be included in the register map used for NPU control, or it can be provided as a separate register map or table.

[0696] ANN DL information can be provided to the processor (i.e., NPU) and the SAM controller, respectively. Furthermore, the ANN DL information provided to the NPU and the SAM controller can be identical to each other, or at least partially identical.

[0697] The SAM controller can be used to send read / write commands to the SAM, which serves as the main memory, based on the sequence information (i.e., sequential access information) in the ANN DL information, and provide the data requested by the processor.

[0698] The SAM, acting as the main memory, can determine the size of the requested data based on the sequence information in the ANN DL information. The ANN DL information can vary depending on the number of PEs in the processor (i.e., NPU), the size of the processor (i.e., NPU) cache, the kernel to be used for the corresponding layer, the size of the feature map, and so on.

[0699] For example, when the data size is larger than the cache size, the processor can use a tiling algorithm. Furthermore, the SAM controller can be configured to operate according to the processor's processing method.

[0700] For example, for delay hiding in main memory, the ANN DL can be determined. In other words, for delay hiding, the ANN DL can also be set so that the cache first has data of a size corresponding to the minimum number of clock cycles.

[0701] When the processor's processing method changes, for example, ANN DL information can be changed based on a weight-stable method, an input-stable method, or an output-stable method.

[0702] Unless there are special circumstances, the SAM according to the examples in this disclosure can be configured to minimize the references above. Figure 17 The frequency of occurrence of the described E2 or E3 delay. That is, unless there are special circumstances, SAM memory operations can be performed to sequentially access rows of memory elements in the memory bank. Therefore, as... Figure 17 The frequency of occurrence of the delays E2 or E3 shown can be minimized.

[0703] In other words, SAM can be configured to operate by sequentially addressing memory elements for each memory operation within a cell of the ANN DL. All columns of memory elements corresponding to a row of a memory element in the SAM can be latched by the corresponding sense amplifier. All data latched in the sense amplifier can be read without incurring additional RAS delay. Therefore, memory elements corresponding to columns of a row can be read sequentially.

[0704] However, this disclosure is not limited thereto; the reading order of the data latched in the sensing amplifier can be changed, and even in this case, there is no separate RAS delay.

[0705] In other words, SAM's sequential addressing technique may mean that when performing memory operations within a cell of an ANN DL, the row and column addresses of memory elements gradually change.

[0706] The SAM controller can be configured to directly control the address of the SAM, which serves as main memory within the cells of the ANN DL. Therefore, the SAM controller can be configured to directly control the RAS and CAS signals used to access the memory elements of the SAM.

[0707] Figure 19 This is an example table showing the computational load and data size for 16 layers.

[0708] exist Figure 19 In the examples, when the artificial neural network model is VGG16, the structural information of the input feature map, output feature map, and kernel for each of the 16 layers is described. Various examples of this disclosure can be configured to generate at least one ANN DL information based on at least one artificial neural network model.

[0709] exist Figure 19 In the table, layers 1 to 13 are layers used for convolution, and layers 14 to 16 include fully connected layers.

[0710] Generally, artificial neural network models should be computed in the order of layers. However, due to various reasons, the number of operation sequences of an artificial neural network model processed by a processor may increase or decrease.

[0711] In theory, a single layer of an artificial neural network model can be processed with a single convolution operation. However, a single convolution operation can be performed multiple times under various conditions. That is, the number of convolution operations can increase to the number of tiles.

[0712] For example, ANN DL information can be changed based on the layer structure of the artificial neural network, the PE array structure of the processor (i.e., NPU), and the size of the processor's internal memory.

[0713] For example, if the kernel's internal memory size is 256KB and the kernel size of layer 1 is 3.2MB, then the number of tiling operations applicable to the kernel's internal memory can be thirteen.

[0714] In addition, the sequence of thirteen tiling operations to be processed by the processor can be determined.

[0715] In other words, the number of steps in ANN DL may vary depending on the size of the processor's internal memory. Therefore, the number of steps in ANN DL can also be increased. On the other hand, if the internal memory size of the input feature map is 256KB and the size of the input feature map of layer 1 is 1.7KB, tiling may not be necessary. Even for the output feature map of layer 1, tiling may not be necessary.

[0716] In other words, when the order of operations of an artificial neural network model processed by a processor changes, the ANN DL information of the artificial neural network model also changes.

[0717] Therefore, the ANN DL information of an artificial neural network model can be configured to include sequence information that changes due to tiling.

[0718] Figure 20 This is an example table showing the computational load and data size for 28 layers.

[0719] exist Figure 20 In the examples, when the artificial neural network model is MobileNet V1.0, the structural information of the input feature map, output feature map, and kernel is described for each of the 28 layers. Various examples of this disclosure can be configured to generate at least one ANN DL information based on at least one artificial neural network model.

[0720] exist Figure 20 In the table shown, layers 1 to 28 include convolutional layers, depth-wise convolutional layers, and point-wise convolutional layers.

[0721] Generally, artificial neural network models should be computed layer by layer. However, due to various reasons, the sequence of operations may change. If the sequence of operations changes, the ANN DL information of the artificial neural network model will also change.

[0722] For example, when a processor processes two artificial neural network models, the ANN DL information processed by one processor can be... Figure 19 and 20 The combination of local information of artificial neural network data for each artificial neural network model shown in a specific sequence.

[0723] For example, when two processors process an artificial neural network model, the ANN DL information processed by the two processors can be separated, making... Figure 19 The ANN DL information of the artificial neural network model shown is processed by two processors.

[0724] Figure 21 This is a table illustrating a first example of accessing memory based on sequence information in Artificial Neural Network Data Locality (ANN DL) information.

[0725] exist Figure 21In the first example shown, when the artificial neural network model is Mobilenet V1.0, the SAM controller can be configured to have ANN DL information including 84 steps to handle 28 layers of computation. That is, the sequential access information can be determined from the perspective of the row and column addresses of the SAM based on the order of each step.

[0726] The SAM, which serves as the main memory, can be configured to operate based on ANN DL information included in the SAM controller.

[0727] Here, ANN DL information refers to the sequence of data processing for an artificial neural network model of a processor, generated by a compiler or SAM controller under at least one of the following conditions.

[0728] a. ANN model structure (VGG16 or Mobilenet V1.0, etc.).

[0729] b. Processor architecture (e.g., depending on the architecture of the CPU, GPU, and NPU). For example, in the case of an NPU, the number of PEs, such as a stationary structure like input stationary, output stationary, or weighted stationary, etc.

[0730] c. Size of the cache (when the size of the cache is smaller than the data size, a tiling algorithm is required, etc.).

[0731] d. Data size for each domain and each layer. For example, a domain may include an input feature map (IFMAP), an output feature map (OFMAP), and a kernel.

[0732] e. Processing strategy.

[0733] f. Data reuse rate. For example, the data request sequence for a specific domain can be determined, such as reading the input feature map IFMAP first or reading the kernel first.

[0734] The strategy can vary depending on the processor architecture or compiler algorithm.

[0735] According to the examples of this disclosure, the SAM controller can set the row and column addresses of the SAM's memory elements to be ordered based on the ANN DL. For example, it can be configured such that the row and column addresses of the SAM's memory elements are ordered within the cells of the ANNDL.

[0736] Figure 22 It is shown in a simplified way Figure 21 The table shown is an example table.

[0737] exist Figure 22 In this text, for ease of interpretation, data size and memory address are represented by symbols.

[0738] refer to Figure 22 As can be seen, the SAM controller can establish an address allocation strategy for the SAM based on the sequence information in the ANN DL information. More specifically, the SAM controller can be configured to directly control the row and column addresses of the SAM's memory elements.

[0739] Based on the sequential access information in the ANN DL information, at least some or all of the data can be stored in the memory within the ANN DL cell. In this case, data storage can be optimized for burst patterns.

[0740] ANN DL information may include, for example, pattern information, in the order of i) reading the input feature map, ii) reading the corresponding kernel, and iii) writing the output feature map. However, this disclosure is not limited to the above-described pattern, and various patterns are disclosed. Furthermore, the pattern can be set differently for each layer.

[0741] At this point, SAM, acting as the main memory, can control the CAS or RAS signals based on the ANN DL information to operate in burst mode. (Reference) Figure 41 or Figure 42 The example illustrates how row address decoders and column multiplexers / demultiplexers can be directly controlled by controlling the CAS signal, RAS signal, and address signal.

[0742] In this scenario, the SAM controller can predict that the processor will request data in a specific sequence based on the sequence information in the ANN DL information.

[0743] The SAM controller can analyze the ANN DL information of the compiled artificial neural network model and directly control the SAM's CAS and / or RAS signals, so that data requested by the processor can be sequentially arranged in the SAM's memory elements. Alternatively, segmented data can be rearranged sequentially. Therefore, the SAM can provide data to the SAM controller sequentially.

[0744] In other words, SAM, as the main memory, can be configured to operate in burst mode from the start address to the end address within the cells of the ANN DL.

[0745] Alternatively, SAM, acting as main memory, can analyze the compiled ANN DL information, arrange the data to be requested by the NPU into sequential addresses, and then provide the data sequentially.

[0746] Therefore, the SAM controller according to the example of this disclosure can set the row and column addresses of the SAM's memory elements in a sequential manner based on the ANN DL.

[0747] Each cell in an ANN DL can have a corresponding data size. For example, the cell in the first ANN DL can have data of size A, and can have a start address and an end address corresponding to size A. Therefore, the SAM's operating mode can be configured to operate in essentially the same way as the burst mode of DRAM, and the SAM can essentially operate in burst mode when the SAM controller generates a read command.

[0748] Furthermore, SAM can operate in de facto burst mode based on ANN DL even when the command from the processor is in read mode rather than read burst mode.

[0749] Furthermore, based on ANN DL, all data can be operated in burst mode. However, this disclosure is not limited to this; most data can also be set to burst mode. That is, at least a portion of the data can be operated on outside of burst mode.

[0750] Figure 23 SAM was shown according to Figure 22 The table shown is an example of setting up a memory address mapping.

[0751] The SAM controller can control the SAM's CAS and / or RAS signals based on compiled ANN DL information, and can arrange the data to be requested by the processor so that it is ordered in the memory map.

[0752] Since SAM already knows the sequence in which the processor will generate read or write commands for a specific size of data at a specific address in memory, the data can be arranged in the following sequence.

[0753] According to such Figure 23 In the example shown, each of the data A through K is stored using sequential addresses. Thus, since the data is stored sequentially based on the ANN DL, burst mode operation is possible at least for each ANN DL cell. Furthermore, according to the examples of this disclosure, burst mode operation of multiple ANN DL cells is also possible because adjacent ANN DL cells can also have sequential addresses. In each of the data A through K, the data bits can be stored sequentially. Therefore, SAM can operate in burst mode. Sequential addresses can refer to the column and row addresses of the sequentially increasing array of memory elements.

[0754] Therefore, each piece of data in an ANN DL cell can be read in burst mode, and data stored at consecutive addresses can also be read in burst mode.

[0755] Preferably, all ANN DL units from #1 to #15 can operate in burst mode, but this disclosure is not limited thereto, and the data of at least one ANN DL unit can be configured to operate in burst mode.

[0756] The process following the setup of memory address mapping based on ANN DL information will be described below.

[0757] ANN DL Unit #1: The processor and / or SAM controller can request the SAM to read data A in read burst mode. In the case of ANN DL Unit #1, since data A is stored sequentially, the SAM can operate in read burst mode until data A is read.

[0758] ANN DL Unit #2: The processor and / or SAM controller can request the SAM to read data B in read burst mode. In the case of ANN DL Unit #2, since the data is stored sequentially, it can operate in read burst mode until data B is read.

[0759] Since data A and B are stored sequentially in the memory address mapping, data A and B, i.e. data of consecutive ANN DL cells, can be operated in read burst mode.

[0760] ANN DL Unit #3: The processor and / or SAM controller can request the SAM to write the output feature map OFMAP C data in write burst mode. Since the C data has a memory address following the B data, it can be written to the SAM in read burst mode.

[0761] ANN DL Unit #4: The processor and / or SAM controller can request the SAM to read the C data again in read burst mode, which is the input feature map IFMAP.

[0762] ANN DL unit #3 and ANN DL unit #4 will be described further. The C data, which serves as the output feature map of the first layer, is reused as the input feature map of the second layer.

[0763] As mentioned above, since artificial neural network models can request a read operation again immediately after a write operation, and the data is the same, refresh operations of memory elements that may occur during write and read operations can be omitted based on ANN DL. Therefore, the time required to refresh C data can be omitted and power consumption can be reduced.

[0764] Furthermore, feature maps of a particular artificial neural network model can have the property of not being reused after their ANN DL sequence has passed through. Therefore, since such feature maps no longer need to be maintained, artificial neural network operations will not fail even if the stored data becomes corrupted due to lack of updates.

[0765] ANN DL Unit #5: The processor and / or SAM controller can request the SAM to read D data in read burst mode. Since the address of D data immediately follows the address of C data, it can be read continuously in read burst mode.

[0766] ANN DL Unit #6: The processor and / or SAM controller can request the SAM to write E data in write burst mode. Since the E data has an address following the address of the D data, it can be written to the SAM sequentially according to the write burst mode.

[0767] ANN DL Unit #7: The processor and / or SAM controller may request the SAM to read the E data again in read burst mode.

[0768] ANN DL units #6 and #7 will be described further. The D data (i.e., the output feature map OFMAP of the second layer) is reused as the input feature map IFMAP of the third layer. Thus, since the artificial neural network model can request a read again immediately after writing the same data, refresh operations that might occur between writes and reads can be omitted based on ANN DL. Therefore, the time required to refresh the E data can be saved, and power consumption can be reduced.

[0769] Furthermore, due to the nature of artificial neural network models, feature maps of a particular artificial neural network model may no longer be reused when the ANN DL sequence passes through. Therefore, since such feature maps no longer need to be maintained, artificial neural network operations will not encounter errors even without refreshing memory elements.

[0770] Figure 24 This is a table illustrating a second example of accessing memory based on sequence information in ANN DL information.

[0771] exist Figure 24 In the second example shown, when the artificial neural network model is MobileNet V1.0, the ANN DL can be configured to read the kernels before the input feature maps. For this purpose, the sequence information in the ANN DL information can include the sequence of kernels read before the input feature maps.

[0772] This second example may be more effective as the data size of the feature map increases.

[0773] After reading the kernel from SAM, convolution can begin as soon as the input feature map IFMAP is received.

[0774] Specifically, the ANN DL information according to the second example may include information about the following sequence patterns: i) first reading the kernel, ii) reading the corresponding input feature map, and 3) writing the output feature map to memory.

[0775] As the main memory, SAM can control the CAS signal and / or RAS signal according to the ANN DL information to operate in burst mode.

[0776] The sequence of data operation requests from the processor is based on a preset ANN DL sequence.

[0777] The SAM controller can control the SAM's CAS and / or RAS signals based on compiled ANN DL information to sequentially allocate data requested by the processor or NPU, and then operate sequentially in burst mode.

[0778] Based on compiled ANN DL information, the SAM controller can perform optimizations to operate SAM in burst mode by sequentially arranging data requested by the processor or NPU.

[0779] and Figure 23 Compared to the first example, even when dealing with the same artificial neural network model, the memory address mapping of the second example may differ from that of the first example.

[0780] Figure 25 SAM was shown according to Figure 24 The table shown is an example of setting up a memory address mapping.

[0781] As the main memory, SAM can control the CAS and / or RAS signals based on the compiled ANN DL information, so that the data to be requested by the processor is arranged in order in the memory address mapping.

[0782] The SAM controller can control the SAM's CAS and / or RAS signals based on the compiled ANN DL information to sequentially arrange the data to be requested by the processor in the memory address map.

[0783] Because the SAM controller knows whether the processor will send a read or write command for a specific size of data to a specific address in memory, the SAM can know in what sequence the data will be processed.

[0784] refer to Figure 25Each of the data A through K is stored according to a sequential memory address. Because the data is ordered in this way, SAM can operate in burst mode at least within the ANN DL cells. Furthermore, according to the examples of this disclosure, burst mode operation of multiple ANN DL cells is also possible because adjacent ANN DL cells can also have sequential addresses.

[0785] Since the data bits in data A through data K are also stored sequentially, the memory can operate in burst mode.

[0786] In other words, the bits that make up each piece of data can also be read or written in burst mode, and since each piece of data is consecutive to the next, it can be read or written in burst mode.

[0787] The following section describes some explanations of the inference steps following the setup of memory mapping (i.e., memory address mapping) based on ANN DL.

[0788] i) The processor and / or SAM controller can request the SAM to read data according to the read burst mode. Since the data is stored sequentially, the read burst mode can be performed while reading data A.

[0789] ii) The processor and / or SAM controller may request the SAM to read B data according to the read burst mode. Since the data is stored sequentially, the read burst mode can be performed simultaneously with reading B data.

[0790] Since data A and B are stored sequentially, they can be manipulated in burst read mode. In other words, continuous data in ANN DL can be manipulated in continuous burst read mode.

[0791] iii) The processor and / or SAM controller may request the SAM to write C data, which is the output feature map, according to the write burst mode. Since the C data has a memory address following the B data, it can be written into memory according to the write burst mode.

[0792] iv) The processor and / or SAM controller may request the SAM to read D data according to the read burst mode.

[0793] v) The processor and / or SAM controller can request SAM to read C data again according to the read burst pattern. That is, the output feature map OFMAP of the previous layer can be used as the input feature map IFMAP of the next layer. Since the processor and / or SAM controller know in advance that SAM is requested to read C data after D data, operations such as precharging and / or refreshing of the corresponding memory elements can be selectively scheduled in advance.

[0794] vi) The processor and / or SAM controller may request SAM to write E data according to the write burst mode. Since the processor and / or SAM controller know in advance that SAM is requested to write E data after C data, operations such as precharge and / or refresh of the appropriate memory elements can be scheduled in advance.

[0795] vii) The processor and / or SAM controller may request the SAM to read the F data according to the read burst mode. Since the address of the F data immediately follows the address of the E data, continuous operation is possible in read burst mode.

[0796] The processor and / or SAM controller can predict in advance requests to the SAM to read again the same data written to ANN DL unit #6 from ANN DL unit #8. Therefore, based on ANN DL, it is possible to predict or calculate when data access requests corresponding to ANN DL units #6 and #8 will be processed. For prediction or calculation, information such as the processor's clock speed, the size of the E-data corresponding to ANN DL units #6 and #8, and the bandwidth of the memory bus can also be utilized. Therefore, the SAM controller or SAM can omit tasks such as precharging and / or refreshing the corresponding memory elements or schedule these tasks with optimal timing.

[0797] Figure 26 This is a table illustrating a third example of accessing memory based on sequence information in ANN DL information.

[0798] exist Figure 26 In the third example shown, a specific region in the memory can be set as a common region of the input feature map and the output feature map. That is, the SAM and / or the SAM controller can be configured to classify regions of the SAM based on a specific domain.

[0799] Since the input feature maps and / or output feature maps may be data that cannot be reused once used, they can be recorded alternately in the same area.

[0800] In the table shown, M_FMAP represents the size of the largest feature map among the multiple input and output feature maps. Since the size of the feature maps differs for each layer, setting a maximum value for the feature maps in an artificial neural network model can prevent problems such as overflow.

[0801] All read or write operations on the feature map can start at the same address, while the end address can be changed depending on the actual size of the corresponding feature map.

[0802] In the third example, since a specific area of ​​the memory is shared, the following conditions must be met.

[0803] M_FMAP≥C, E, G, I and K ( Figure 26 The middle part is omitted, but the example ANN DL information of Mobilenet V1.0 includes 84 units (M_FMAP is the maximum value in all feature maps). In memory, the kernel can store them sequentially.

[0804] Figure 27A and 27B An example of setting up memory address mapping based on ANN DL information is shown.

[0805] The SAM controller can control the SAM's CAS and / or RAS signals based on compiled ANN DL information to sequentially allocate data requested by the processor (e.g., NPU) in the memory address map.

[0806] like Figure 27A and 27B As shown, since SAM knows that the processor (e.g., NPU) will command to read or write data of a specific size to a specific address in memory, SAM knows in what sequence the data will be processed.

[0807] Feature maps can be recorded in overwrite commands for common areas in memory, and the kernel uses sequentially arranged memory addresses to store them. Therefore, continuous data can be read or written according to burst patterns.

[0808] The first to third examples described so far will be briefly described below.

[0809] Referring to the first to third examples, memory address mapping can be set based on ANN DL information. Memory address mapping can be set according to various conditions, performance, algorithms, and the structure of the ANN model, as described above. Furthermore, SAM can set memory address mapping based on ANN DL information so that data can be read or written in burst mode using the ANN DL information.

[0810] Based on the first to third examples, kernel performance improvements can be achieved due to the sequential nature of SAM.

[0811] On the other hand, feature map operations can be repeated in the order from write operations to read operations.

[0812] When the main memory uses a DRAM memory element structure, due to the inherent characteristics of DRAM, once data is read from the memory element, the charge in the capacitor of the memory element will discharge, resulting in data loss. Therefore, a recovery operation can be performed to recharge the capacitor. In other words, multiple dynamic memory elements of the main memory may have leakage current characteristics.

[0813] When the main memory uses a DRAM memory element structure, due to the inherent characteristics of DRAM, once data is read from the memory element, the charge in the capacitor of the memory element will discharge, resulting in data loss. Therefore, a recovery operation can be performed to recharge the capacitor.

[0814] The first example is for a sequence pattern, but since the input feature map IFMAP is read from memory first, the convolution operation can be performed after the kernel is read.

[0815] In the second example, since the kernel is read from memory first, the convolution operation can begin after reading the input feature map IFMAP. From the perspective of performing convolution, the second example has an advantage.

[0816] In the third example, it may be effective when the main memory capacity is relatively small. Alternatively, it may be effective when the feature map and kernel are separated, such as when the main memory is a two-channel memory.

[0817] Since DRAM buses are typically single-channel, data can be sent and received as in the first to third examples. However, in another example, using multiple memories or multiple channels, SAM can also be implemented by differentiating weights and feature maps.

[0818] However, the examples in this disclosure are not limited thereto, and Figures 22 to 2 The examples described in 7 can be set differently for each layer of an artificial neural network model, depending on the size of the feature maps and the kernel of each layer.

[0819] Figure 28 This is a conceptual diagram illustrating the control signals of the SAM controller.

[0820] Figure 28 The memory, SAM controller, and processor are shown. Signals transmitted from the processor may not be transmitted over separate physical lines, but may be logical signals (i.e., data packets) transmitted over one or more lines. However, this disclosure is not limited thereto.

[0821] The SAM controller may include internal memory for storing ANN DL information.

[0822] ANN DL information may include information compiled to utilize a processor (e.g., an NPU).

[0823] Read / Write Command: This refers to the read or write command signal transmitted based on the sequence information in the ANN DL information. The memory address corresponding to each command signal can be sent along with the memory's start and end addresses or clock count information.

[0824] Sideband signals: can selectively include various control signals to improve processing efficiency based on ANN DL information, as needed.

[0825] The RESET signal can be used to reset the memory address mapping when the ANN model changes.

[0826] ENABLE signal: When the ENABLE signal is ON, data can be transmitted to the processor.

[0827] ANN DL information and SIDEBAND SIGNAL may contain some redundant signals. However, depending on the artificial neural network architecture, ANN DL information can be static information, while the sideband signals can be dynamic control signals used for ANN operations.

[0828] Figure 29 The diagram is based on Figure 28 The example diagram shown illustrates how sideband signals set up memory address mapping.

[0829] like Figure 29 As shown, the memory address mapping can be configured to handle multiple ANN models.

[0830] When a processor (e.g., an NPU) performs time-division multiplexing operations according to the # number order of the ANN MODEL, the memory address mapping can be set sequentially according to the # number order of the ANN MODEL. Therefore, SAM can operate in burst mode when the ANN model changes. The memory address mapping for each ANN model can be set according to the first to third examples above.

[0831] Figure 30A An example of setting memory address mapping based on sideband signals is shown. Figure 30B An example is shown in which only the kernel's memory address mapping is set sequentially.

[0832] like Figure 30A As shown, multiple threads can be set up for a specific ANN model (e.g., ANN model #1). That is, when multiple users access the model, the memory address mapping can be configured to process the ANN model in multiple threads.

[0833] By using multiple threads, multiple users can share the kernel of a single ANN model. Each thread can be allocated a memory address mapping for storing the input feature map and / or the output feature map.

[0834] Alternatively, when using multiple threads, the kernel can be mapped sequentially into the memory address map, such as... Figure 30BAs shown, the number of additional M_FMAPs can be created as the number of threads increases.

[0835] In other words, regardless of the number of users, the kernel's memory address can be shared, and the memory address of the feature map can be configured to increase proportionally with the number of users.

[0836] Figure 31A This is an example diagram illustrating the READ_DISCARD command transmitted via sideband signals according to an example of this disclosure, and Figure 31B An example of the READ command is shown.

[0837] Figure 31B The name t shown RAS Represents data sensing (t) RCD + Data recovery time to DRAM components.

[0838] Data sensing (t) RCD The latching time refers to the time it takes to latch data to the sensing amplifier. For latching operations, precharging, access, and sensing operations may be required.

[0839] For the above operations, please refer to Figure 32 , 33 And 34.

[0840] In addition, you can refer to Figure 17 To describe this example.

[0841] Based on the example provided here, it is possible to do so only in t RCD The READ_DISCARD command is executed during this period, and the operation to restore data to the DRAM element may not be performed.

[0842] Therefore, the latency and power required to restore data to DRAM cells can be reduced. For example, according to Figure 25 After ANN DL cell #3 in the ANN DL information writes C data to memory, and then reads C data again from memory in ANN DL cell #5, the C data will no longer be used according to the ANN DL information. Therefore, there is no need to perform the operation of restoring the data to the DRAM cell. Therefore, the operation can be determined using sequence information and / or domain information.

[0843] For example, the READ_DISCARD command can be set to a specific ANN DL cell #.

[0844] For example, the output feature map (OFMAP) of ANN DL unit #3 can be reused as the input feature map (IFMAP) of the next layer (i.e., ANN DL unit #5). In other words, the input feature map (IFMAP) can take advantage of the fact that the input feature map is not reused after convolution with kernels.

[0845] In other words, when reading the input feature map (IFMAP), the READ_DISCARD command can be set to the corresponding ANN DL unit #.

[0846] For example, when the output feature map (OFMAP) of the first layer is written to memory, the corresponding data is used as the input feature map (IFMAP) of the second layer and read from memory. However, since the input feature map (IFMAP) is not reused, even if data is lost due to the failure to restore the data to the DRAM cell, the ANN operation will not be affected. Therefore, according to the examples of this disclosure, the SAM controller can be configured to instruct a READ-DISCARD command to the memory.

[0847] This principle is Figure 31A As shown in the figure, Figure 31A The data sensing shown (t) RCD () is the time it takes for the sensing amplifier to read the value stored in the memory element of a specific row.

[0848] In other words, the READ_DISCARD command can be executed within a cell of a memory row.

[0849] "Recovering data to a DRAM cell" refers to the operation of using a sense amplifier to recover latched data back to the memory element when data stored in the memory element is lost due to a read operation performed using a sense amplifier.

[0850] Figure 32 A portion of a circuit diagram of an exemplary SAM implemented as a memory element of DRAM according to an example of this disclosure is shown.

[0851] Figure 32 The circuit diagram of the SAM shown includes a sensing circuit (i.e., a sensing amplifier). The sensing circuit of the SAM amplifies the difference between the reference voltage Vref provided to the bit line and the voltage on the bit line to generate a digital signal 0 or 1.

[0852] SAM's sensing circuitry can selectively recover charge from discharged memory elements via bit lines. A recovery operation is also performed when a READ command is executed. In the case of a READ-DISCARD command, the recovery operation may not be performed.

[0853] The sensing circuit of SAM can act as a buffer memory for latching the sensed voltage.

[0854] Here, the capacitors of the memory element may have leakage current characteristics.

[0855] Figure 33 It is used for explanation Figure 32 An example diagram of the pre-charge operation in a SAM circuit diagram.

[0856] During the precharge operation, an equalization signal EQ is provided, and VOLTAGE EQ.CIRCUIT is activated. When the EQ signal is provided, a voltage Vref = Vcc / 2 is applied to the bit line through each Tr and (Reversed bit line). Therefore, bit line and Charged with Vref voltage. Additionally, the voltage equalization circuit (VOLTAGE EQ.CIRCUIT) disconnects the bit line and... Make the two lines have the same voltage.

[0857] Figure 34 It is used for explanation Figure 32 The diagram shown is an example of memory element access operations in the SAM circuit diagram.

[0858] During the access operation, the bit lines can be charged in the following order.

[0859] i) Using the voltage Vcc + Vt (in Figure 34 The diagram shows a thick word line (the word line is overdriven to correspond to the precharge bit line to be accessed). A voltage of Vcc + Vt is supplied to the word line to turn on the transistor Tr of the memory element to be accessed. When Tr is off, data output is blocked.

[0860] ii) When the value stored in the capacitor of the memory element Tr is 1 (in Figure 34 When the bit line (shown as a thick bit line) is in the middle, the capacitor discharges, and the voltage of the bit line rises from Vref to Vref+. At this time, the voltage of the capacitor decreases due to discharge, and the stored data is lost.

[0861] Figure 35 It is used for explanation Figure 32 The diagram shown is an example of data sensing operation in the SAM circuit diagram.

[0862] During sensing operation, the sensing circuit charges the voltage of the bit line in the following sequence.

[0863] i) When the bit line is charged with Vref+ voltage (in) Figure 35 (Seen in thick solid line in the middle), the transistor in the lower left corner of the sensing circuit is turned on. Therefore, the ground voltage GND, i.e., SAN, is applied to... therefore, This becomes the GND voltage. Therefore, the upper right transistor turns on.

[0864] ii) such as Figure 35 As shown by the thick line, when the upper right transistor of the sensing circuit is turned on, SAP (i.e., the Vcc voltage) is applied to the bit line. Additionally, the voltage Vcc from Vref+ is applied to the gate of the lower left transistor. Therefore, the applied bit line is 3V.

[0865] iii) The data in the DRAM is now ready to be read. When the CSL signal is provided, the output of the sensing circuit can be generated.

[0866] Figure 36 It is used for explanation Figure 32 The diagram shows an example of the READ-DISCARD operation in the SAM circuit diagram.

[0867] In READ-DISCARD operation, a column select line signal CSL is provided, and the sensing circuit outputs data accordingly.

[0868] According to the example presented herein, after reading data stored in a memory element, no recovery process for recharging the corresponding memory element is performed, thereby reducing power consumption and recovery time. This is applicable, for example, when storing an output feature map and then using it as an input feature map for the next layer.

[0869] refer to Figure 31A and Figure 31B The comparison can be made through t RAS Time reduces memory latency.

[0870] At the same time, to prevent the capacitor from being charged, the overdrive word line driven by VCC+Vt can be turned off. Therefore, power consumption can be reduced.

[0871] Figure 37 It is used for explanation Figure 32 Example diagram of the READ operation in the SAM circuit diagram.

[0872] According to the READ command, the CSL signal is provided and the sensing circuit outputs data.

[0873] Based on the example presented here, after reading the data stored in the memory element, the charge can be restored by recharging the corresponding memory element.

[0874] refer to Figure 31B Recovery requires t RAS time.

[0875] Holding a word line active allows the fully driven bit line voltage to be restored via the corresponding transistor in the sensing circuit and the capacitor. Therefore, the power consumption during restoration needs to be considered.

[0876] Figure 38A This is an example waveform diagram of the READ-DISCARD operation. Figure 38B This is an example waveform diagram of the READ operation.

[0877] and Figure 38B Conversely, refer to Figure 38A Since there is no RESTORE operation, the tRAS time can be shortened. See reference... Figure 31B and 31B In order to understand the differences between the corresponding operations.

[0878] Figure 39 It means Figure 21 A portion of the table is used to explain REFREASH operations.

[0879] Figure 39 The table is used to conceptually explain the time required to perform one inference operation using an ANN model.

[0880] The time required for each ANN DL# cell can be measured, calculated, or predicted based on factors such as processor processing speed, data bus bandwidth, and memory operation speed.

[0881] The SAM controller can control whether to refresh data in a specific region (e.g., a domain) of the memory based on ANN DL information. The SAM controller can also measure inference time based on ANN DL information. For example, the repetition time of the same ANNDL cell # in the ANN DL information can be measured. That is, after performing an operation on ANN DL cell #1, the time it takes for the operation on ANN DL cell #1 to return can be measured. As another example, the time spent executing start cell # and end cell # in the ANN DL information can be measured. That is, the time spent executing operations from cell #1 to cell #84 in the ANN DL information can be measured. As another example, processing time can be measured by setting a specific time period in the ANN DL information.

[0882] If the SAM controller determines that an inference operation has been completed within a threshold time, memory refresh can be disabled. For example, if an inference operation is completed within a threshold time based on ANN DL information, memory region refresh for the memory kernel can be disabled.

[0883] The reasons for this are as follows. When an inference operation is completed based on ANN DL information, a "read" or "write" to all kernels of the ANN model is performed once in memory. In the case of DRAM memory element structures, a "read" is essentially the same as a refresh, so data can be retained even if refresh is not performed repeatedly. However, when inference is interrupted during inference or a threshold time is exceeded, the SAM controller may only refresh the memory elements (e.g., rows) storing the kernels. As another example, if an inference operation is completed within a threshold time based on ANN DL information, the SAM controller may disable refresh only for the memory region storing the feature map. The reasons for this are as follows. Since feature maps cannot be reused, they are not sensitive to data loss, so refresh can be disabled. Furthermore, when a read-discard operation is performed, disabling refresh may be effective because data has already been lost.

[0884] In detail, since the kernel can be a fixed value, it can be refreshed periodically. However, as mentioned above, if the repetition of kernel read operations can be predicted within a threshold time based on ANN DL, then refreshing can be disabled.

[0885] However, the examples disclosed herein are not limited thereto, and the read, write, and read-discard commands of SAM can be appropriately selected based on ANNDL, taking into account the characteristics of the data, processing time, reuse, etc.

[0886] Meanwhile, the aforementioned threshold time can be exemplarily set to the refresh threshold time RT. th = 32ms to 64ms. The recommended time to prevent data loss may vary depending on the capacitance and leakage current characteristics of the memory elements.

[0887] Furthermore, when the inference time (IT) is satisfied... <RT th At that time, the memory element storing the feature map may not receive a refresh command.

[0888] Data refresh strategies can be set differently based on ANN DL information.

[0889] For example, in the case of the kernel, a refresh policy can be set to increase the level of data protection, while in the case of the feature map, a refresh policy can be set to decrease the level of data protection.

[0890] Disabling refresh based on ANN DL information can reduce memory operation latency and lower power consumption.

[0891] When the data of an artificial neural network model is distributed and stored in multiple memory banks, the precharge time of each memory bank can be controlled based on ANNDL information.

[0892] Figure 40 Examples of SAM memories implemented in various forms according to the present disclosure are shown.

[0893] SAM memory can be implemented in various forms depending on the application area.

[0894] The memory bus, serving as the data transfer path between the cache memory and / or processor in the AMC, can be implemented as a single-channel or dual-channel bus. While power consumption increases with the number of channels, one advantage is that bandwidth can be improved by managing the core and / or feature maps separately. Using two channels doubles the bandwidth compared to using one channel, allowing more data to be transferred to the cache and / or processor in the AMC. Operation can be controlled based on ANN DL.

[0895] Multiple SAM memories can be grouped and driven by “level”.

[0896] Each SAM memory may include a memory bank, which is a collection of independently operating arrays of memory elements. For example, a memory bank may include eight arrays of memory elements. Interleaving multiple memory banks can implement a high-bandwidth memory bus using low-bandwidth devices. Each memory array may include a row decoder, a column decoder, a sense amplifier, and an input / output buffer. A "row" refers to a row of memory elements in the memory array. A "column" refers to a column of memory elements in the memory array.

[0897] Figure 41 This is an example diagram illustrating a method for mapping main memory addresses based on ANN DL information.

[0898] refer to Figure 41 The basic structure of a SAM is shown. A SAM comprises multiple memory elements in a matrix structure with row and column addresses. A SAM can be implemented as, for example, DRAM. However, the examples in this disclosure are not limited thereto.

[0899] The sense amplifier is positioned at the bottom of multiple memory cells in the matrix structure. The row address decoder selects a specific row. Therefore, a RAS delay is required to perform the corresponding operation. The data of the memory element in the selected row is latched in the sense amplifier. The column address decoder selects the necessary data from the data latched in the sense amplifier and transfers it to the data buffer. Therefore, a CAS delay is required to perform the corresponding operation. This structure can be referred to as a DRAM memory bank. DRAM can include multiple memory banks.

[0900] In this scenario, when DRAM operates in burst mode, data is read or written simultaneously as the addresses of memory elements increment sequentially. Therefore, RAS latency and CAS latency are minimized compared to reading segmented address data.

[0901] In other words, even if the AMC or NPU instructs burst mode to the main memory, if the data stored in DRAM is actually segmented, the RAS latency and CAS latency are just as significant as the segmentation itself. Therefore, if data is segmented, it is difficult to substantially reduce RAS latency and CAS latency simply by executing burst mode commands.

[0902] In contrast, in the case of SRAM, whether the data is segmented or not has virtually no impact on latency. Therefore, in buffers or internal memory composed of SRAM, the latency caused by data segmentation may not be fatal.

[0903] refer to Figure 41 Based on the ANN DL information, the memory address mapping is set up by considering the order and size of the data requested by the NPU from the DRAM memory elements. The memory mapping can be set based on each data size, and on the start and end addresses. Therefore, if memory operations are performed in the order of the ANN DL information in the SAM, all memory operations can be performed in burst mode.

[0904] therefore, Figure 41 The main memory shown can be controlled according to the memory address and operating mode shown in Table 1.

[0905] Table 1

[0906]

[0907]

[0908] More specifically, for the fields in Table 1, references can also be used. Figure 12 The described domain information. More specifically, it can also utilize... Figure 12 The operating mode information described herein is used for the operating modes in Table 1.

[0909] Because the data is mapped to sequential addresses based on ANN DL information, it can be processed using burst mode commands.

[0910] In other words, AMC can cache the necessary data before the NPU issues a request based on ANN DL information (ANN DL) and can determine the sequence of all requests. Therefore, the cache hit rate of AMC's buffer memory can theoretically reach 100%.

[0911] In addition, since the memory mapping of the main memory is set based on ANN DL information, all memory operations can be performed in burst mode.

[0912] although Figure 29The example illustrates a single memory bank, thus allowing address mapping to be performed using a bank interleaving method based on the configuration of the memory bank, level, and channel.

[0913] Without considering ANN DL information, it is practically impossible to store the data requested by the NPU sequentially in DRAM. In other words, even with general artificial neural network model information, it is impossible to know the sequence of all data operations the NPU requests from main memory without the ANN DL information described in the various examples.

[0914] If the AMC lacks ANN DL information, it has difficulty determining whether the NPU should request the kernel of the first layer of the ANN model first or the input feature map of the first layer first. Therefore, setting up memory address mappings in main memory is extremely difficult, especially considering burst patterns.

[0915] Figure 42 This is an example diagram illustrating another example of a method for mapping main memory addresses based on ANN DL information.

[0916] because Figure 42 The structure of the main memory shown is similar to Figure 41 The main memory shown is basically the same, so redundant descriptions can be omitted.

[0917] refer to Figure 42 Memory mapping can be set based on ANN DL information, taking into account the sequence and size of data requested by the NPU from the memory elements of DRAM. Memory mapping can be set based on each data size, and on the start and end addresses. Therefore, if memory operations are performed in DRAM according to the sequence of ANN DL information, all memory operations can be performed in burst mode.

[0918] therefore, Figure 42 The main memory shown can be controlled based on the memory addresses and operating modes shown in Table 2.

[0919] Corresponding to Figure 42 The ANN data locality information in Table 2 and ANN DL are examples of cases where the NPU is set to share the input feature map and the output feature map.

[0920] Table 2

[0921] ...

Claims

1. A memory device for an artificial neural network (ANN), the memory device comprising: At least one memory element array having N columns and M rows; as well as A memory controller is configured to sequentially perform read or write operations on the data of the at least one memory element array in a burst mode based on predetermined sequential access information from the artificial neural network. In this context, the address mappings of multiple parameters across multiple layers of the artificial neural network are set sequentially, and... The predetermined sequential access information of the artificial neural network is determined based on the characteristics of the processor when the artificial neural network is compiled for execution by the processor. The at least one memory element array includes a plurality of memory banks configured to enable interleaving operations, and The memory controller is further configured to divide and store the data of the artificial neural network in each of the plurality of memory banks based on the predetermined sequential access information, in order to operate in burst mode corresponding to the interleaved operation.

2. The memory device of claim 1, wherein each of the at least one array of memory elements comprises a plurality of dynamic memory elements having leakage current characteristics.

3. The memory device of claim 1, wherein each of the at least one memory element array comprises: A column decoder, which controls access to the N columns; Multiple bit lines connected to the column decoder; A line decoder, which controls access to the M lines; Multiple word lines connect to the line decoder; as well as A sensing amplifier, which is connected to one end of each of the plurality of bit lines.

4. The memory device according to claim 1, The at least one memory element array stores the data required for the operation of the artificial neural network. The memory controller is further configured to control data communication between the processor and the at least one memory element array, and in, The processor is configured to process the artificial neural network operations based on the predetermined sequential access information.

5. The memory device according to claim 1, wherein, The memory controller is also configured to directly control the addresses of the N columns and M rows of the at least one memory element array, such that the at least one memory element array operates sequentially in burst mode based on the predetermined sequential access information.

6. The memory device according to claim 1, wherein, The memory controller is also configured to set the memory address of data for each operational step to be stored in the at least one memory element array based on the sequential access information.

7. The memory device according to claim 1, wherein, The memory controller is also configured to store the data of the artificial neural network by sequentially assigning addresses corresponding to the N columns and M rows of the at least one memory element array.

8. A memory device for an artificial neural network (ANN), the memory device comprising: At least one memory element array, the at least one memory element array comprising a plurality of memory banks configured to enable interleaved operation; and A memory controller is configured to directly control read or write operations of the at least one memory element array based on the ANN data locality information of the artificial neural network. The ANN data locality information of the artificial neural network is determined based on the characteristics of the processor when the artificial neural network is compiled for execution by the processor. The memory controller is further configured to divide and store the data of the artificial neural network in each of the plurality of memory banks based on the ANN data locality information, in order to operate in burst mode corresponding to the interleaved operation. The ANN data locality information includes sequence information used to process data access requests from the ANN.

9. The memory device of claim 8, wherein the ANN data locality information includes predetermined operation sequence information of the artificial neural network.

10. The memory device according to claim 8, wherein, The ANN data locality information includes the data size information of each operation in the preset operation sequence.

11. The memory device according to claim 8, in, The memory controller is also configured to store memory mapping, and The memory mapping is configured sequentially based on operation sequence information and the data size of each operation sequence.

12. The memory device according to claim 8, The ANN data locality information includes signals used to identify weights, input feature maps, and output feature maps. in, The pattern of the operation sequence of the weights, the input feature map, and the output feature map is determined by compilation based on the characteristics of the processor.

13. The memory device of claim 8, wherein the ANN data locality information is determined based on at least one of the characteristics of the artificial neural network model, the characteristics of the processor, the size of the cache memory, and the operation algorithm strategy.

14. A memory device for an artificial neural network (ANN), the memory device comprising: At least one dynamic memory element array, the at least one dynamic memory element array comprising a plurality of memory banks configured to enable interleaving operation; and A memory controller configured to store the artificial neural network data in the at least one dynamic memory element array according to a sequence based on ANN data locality information. The ANN data locality information is determined based on the characteristics of the processor when the artificial neural network is compiled for execution by the processor. The memory controller is further configured to divide and store the data of the artificial neural network in each of the plurality of memory banks based on a sequence of ANN data locality information, in a burst mode corresponding to the interleaved operation. The ANN data locality information includes sequence information used to process data access requests from the ANN.

15. The memory device according to claim 14, wherein, The sequence based on the ANN data locality information includes a repeating pattern with an order of input feature map, kernel, and output feature map.

16. The memory device according to claim 14, in, The ANN data locality information is configured within the unit requested by the processor and sent to the memory controller for data access requests, and The ANN data locality information includes sequence information about all data access requests required to perform the inference operation of the artificial neural network.

17. The memory device of claim 14, wherein, The memory controller is also configured to divide each of the at least one dynamic memory element array into a kernel region and a feature map region based on information used to identify the kernel, input feature map, and output feature map.

18. The memory device of claim 14, further comprising a processor configured to provide the ANN data locality information to the memory controller.

19. The memory device of claim 14, further comprising a processor configured to provide the memory controller with information for identifying the input feature map, the kernel, and the output feature map.

Images