Neural network acceleration system and operation method thereof

Abstract

Disclosed are a neural network acceleration system and an operation method thereof. The neural network acceleration system comprises: a first memory module, based on a first segment of a first embedding and a second segment of a second embedding, generating a first reduced embedding segment through tensor operation; a second memory module, based on a third segment of the first embedding and a fourth segment of the second embedding, generating a second reduced embedding segment through the tensor operation; and a processor, processing the reduced embedding based on a neural network algorithm, the reduced embedding comprising the first reduced embedding segment and the second reduced embedding segment.

Description

[0001] This application claims the priority benefit of Korean Patent Application No. 10-2019-0092337 filed on Jul. 30, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0002] Embodiments of the inventive concept described herein relate to an acceleration system, and more particularly, to a neural network acceleration system and an operating method thereof. Background Art

[0003] A neural network acceleration system is a computing system that processes data based on artificial intelligence / machine learning / deep learning algorithms. A neural network acceleration system can learn input data to produce embeddings, and can perform inference and training operations through embeddings. Neural network acceleration systems using embeddings can be used for natural language processing, advertising, recommendation systems, speech recognition, etc.

[0004] The neural network acceleration system may include a processor for performing inference and training operations using embeddings. Since the data size of the embeddings is very large, the embeddings may be stored in a high-capacity memory outside the processor. The processor may receive the embeddings from the memory outside the processor to perform inference and training operations. In order to quickly perform inference and training operations, the embeddings stored in the memory need to be quickly transferred to the processor. That is, the embedding-based neural network acceleration system requires high-capacity memory and high memory bandwidth. Summary of the invention

[0005] Embodiments of the inventive concept provide a neural network acceleration system capable of providing high-capacity memory and high memory bandwidth and an operating method thereof.

[0006] According to an embodiment of the present invention, a neural network acceleration system includes: a first memory module, which generates a first reduced embedding segment through tensor operation based on a first segment of a first embedding and a second segment of a second embedding; a second memory module, which generates a second reduced embedding segment through tensor operation based on a third segment of the first embedding and a fourth segment of the second embedding; and a processor, which processes the reduced embedding based on a neural network algorithm, the reduced embedding including the first reduced embedding segment and the second reduced embedding segment.

[0007] According to an embodiment, the first embedding may correspond to a first object of a particular category, and the second embedding may correspond to a second object of the particular category.

[0008] According to an embodiment, the first memory module may include: at least one memory device storing the first segment and the second segment; and a tensor operator performing the tensor operation based on the first segment and the second segment.

[0009] According to an embodiment, the at least one memory device may be implemented as a dynamic random access memory.

[0010] According to an embodiment, the size of the first segment may be the same as the size of the third segment.

[0011] According to an embodiment, a data size of the reduced embedding may be smaller than a total data size of the first embedding and the second embedding.

[0012] According to an embodiment, the tensor operation may include at least one of an addition operation, a subtraction operation, a multiplication operation, a concatenation operation, and an average operation.

[0013] According to an embodiment, the neural network acceleration system may further include: a bus, transmitting the first reduced embedding segment from the first memory module and the second reduced embedding segment from the second memory module to the processor based on a preset bandwidth.

[0014] According to an embodiment, the first memory module may be further configured to collect the first segment and the second segment in a storage space corresponding to consecutive addresses, and the first reduced embedded segment may be generated based on the collected first segment and the second segment.

[0015] According to an embodiment of the present invention, a neural network acceleration system includes: a first memory module, which generates a first reduced embedding segment through tensor operation based on a first segment of a first embedding and a second segment of a second embedding; a second memory module, which generates a second reduced embedding segment through tensor operation based on a third segment of the first embedding and a fourth segment of the second embedding; a main processor, which receives the first reduced embedding segment and the second reduced embedding segment through a first bus; and a dedicated processor, which processes the reduced embedding based on a neural network algorithm, the reduced embedding including the first reduced embedding segment and the second reduced embedding segment transmitted through a second bus.

[0016] According to an embodiment, the first embedding may correspond to a first object of a particular category, and the second embedding may correspond to a second object of the particular category.

[0017] According to an embodiment, the first memory module may include: at least one memory device storing the first segment and the second segment; and a tensor operator performing the tensor operation based on the first segment and the second segment.

[0018] According to an embodiment, the first bus may be configured to transfer the first reduced embedded segment and the second reduced embedded segment from the first memory module and the second memory module to the main processor based on a first bandwidth, respectively, and the second bus may be configured to transfer the first reduced embedded segment and the second reduced embedded segment from the main processor to the dedicated processor based on a second bandwidth.

[0019] According to an embodiment, the main processor may also be configured to store the first segment generated by splitting the first embedding and the second segment generated by splitting the second embedding in the first memory module, and may also be configured to store the third segment generated by splitting the first embedding and the fourth segment generated by splitting the second embedding in the second memory module.

[0020] According to an embodiment, the main processor may be further configured to split the first embedding so that the data size of the first segment is the same as the data size of the third segment, and the main processor may be further configured to split the second embedding so that the data size of the second segment is the same as the data size of the fourth segment.

[0021] According to an embodiment, the dedicated processor may include at least one of a graphics processing device and a neural network processing device.

[0022] According to an embodiment, the first memory module may be further configured to collect the first segment and the second segment in a storage space corresponding to consecutive addresses, and the first reduced embedded segment may be generated based on the collected first segment and the second segment.

[0023] According to an embodiment of the present invention, a method for operating a neural network acceleration system (including a first memory module, a second memory module and a processor) includes: the processor stores a first segment generated by splitting a first embedding and a second segment generated by splitting a second embedding in the first memory module, and the processor stores a third segment generated by splitting the first embedding and a fourth segment generated by splitting the second embedding in the second memory module; the first memory module generates a first reduced embedding segment through tensor operations based on the first segment and the second segment, and the second memory module generates a second reduced embedding segment through tensor operations based on the third segment and the fourth segment; and the processor processes the reduced embedding based on a neural network algorithm, the reduced embedding including the first reduced embedding segment and the second reduced embedding segment.

[0024] According to an embodiment, the first embedding may correspond to a first object of a particular category, and the second embedding may correspond to a second object of the particular category.

[0025] According to an embodiment, the step of generating, by the first memory module, the first reduced embedded segment may include: collecting, by the first memory module, the first segment and the second segment in a storage space corresponding to consecutive addresses; and generating the first reduced embedded segment based on the collected first segment and the second segment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects and features of the present inventive concept will become apparent by describing in detail exemplary embodiments of the present inventive concept with reference to the attached drawings.

[0027] Figure 1 is a block diagram illustrating a neural network acceleration system according to an embodiment of the inventive concept.

[0028] Figure 2 is a block diagram illustrating a neural network acceleration system according to another embodiment of the inventive concept.

[0029] Figure 3 is a flowchart describing the operation of a neural network acceleration system according to an embodiment of the inventive concept.

[0030] Figure 4 is a diagram illustrating an embedding segment according to an embodiment of the inventive concept.

[0031] Figure 5 is a diagram illustrating reduced embedding according to an embodiment of the inventive concept.

[0032] FIG. 6A to FIG. 6D is a block diagram illustrating a memory module according to an embodiment of the inventive concept.

[0033] Figure 7 It is shown FIG. 6A to FIG. 6D A block diagram of an example of a buffer device.

[0034] Figure 8 is a block diagram illustrating an extended example of a neural network acceleration system according to an embodiment of the inventive concept. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the inventive concept will be described clearly and in detail so that those skilled in the art can easily implement the inventive concept.

[0036] Figure 1 is a block diagram showing a neural network acceleration system according to an embodiment of the inventive concept. Figure 1, the neural network acceleration system 1000 may include a first memory module 110 to an nth (n is a natural number greater than or equal to 1) memory module 1n0, a main processor 200, a dedicated processor 300, a first bus 1001 and a second bus 1002. For example, the neural network acceleration system 1000 may be implemented as one of a desktop computer, a laptop computer, an embedded system, a server, a car, a mobile device, and an artificial intelligence system.

[0037] Each of the memory modules 110 to 1n0 may operate under the control of the main processor 200. In an exemplary embodiment, each of the memory modules 110 to 1n0 may write data provided from the main processor 200 into an internal memory, or may output data stored in the internal memory and may transmit the data to the main processor 200. In this case, each of the memory modules 110 to 1n0 may transfer data with the main processor 200 through the first bus 1001.

[0038] Each of the memory modules 110 to 1n0 may include a volatile memory such as a dynamic random access memory (DRAM) and a nonvolatile memory such as a flash memory, a phase change memory (PRAM), etc. For example, each of the memory modules 110 to 1n0 may be implemented using an RDIMM (Registered DIMM), LRDIMM (Load Reduced DIMM), NVDIMM (Non-Volatile DIMM) type, etc. based on a dual in-line memory module (DIMM) standard. However, the inventive concept is not limited thereto, and each of the memory modules 110 to 1n0 may be implemented as a semiconductor package having various form factors.

[0039] Figure 1 The memory modules 110 to 1n0 in FIG. 1 are shown as “n” modules, but the neural network acceleration system 1000 may include one or more memory modules.

[0040] The main processor 200 may include a central processing unit (CPU) or an application processor that controls the neural network acceleration system 1000 and performs various operations. For example, the main processor 200 may control the memory modules 110 to 1n0 and the dedicated processor 300.

[0041] The main processor 200 may store codes required for performing neural network-based operations and data accompanying the operations in the memory modules 110 to 1n0. For example, the main processor 200 may store input data including parameters, data sets, etc. associated with the neural network in the memory modules 110 to 1n0.

[0042] The dedicated processor 300 may perform inference and training operations based on various neural network algorithms under the control of the main processor 200. Therefore, the dedicated processor 300 may include an operator or accelerator that performs various operations. For example, the dedicated processor 300 may be implemented as one of the operating devices that perform neural network-based operations, such as a graphics processing unit (GPU) or a neural processing unit (NPU).

[0043] The dedicated processor 300 may transmit data with the main processor 200 through the second bus 1002. For example, the dedicated processor 300 may receive data stored in the memory modules 110 to 1n0 through the main processor 200. The dedicated processor 300 may perform inference and training operations based on the received data. The dedicated processor 300 may send data generated based on the inference and training operations to the main processor 200.

[0044] The first bus 1001 may provide a channel between the memory modules 110 to 1n0 and the main processor 200. The bandwidth of the first bus 1001 may be determined by the number of channels. For example, the first bus 1001 may be based on one of various standards (such as Peripheral Component Interconnect Express (PCIe), Non-Volatile Memory Express (NVMe), Advanced eXtensible Interface (AXI), ARM Microcontroller Bus Architecture (AMBA), NVLink, etc.).

[0045] The second bus 1002 may transfer data between the main processor 200 and the dedicated processor 300. For example, the second bus 1002 may be based on one of various standards such as PCIe, AXI, AMBA, NVLink, and the like.

[0046] In an exemplary embodiment, the main processor 200 may store the embedding in the memory modules 110 to 1n0. In this case, the embedding is a value in which the input data is converted into a vector or multidimensional tensor form by learning, and may indicate information of a specific object in a specific category. For example, the embedding may correspond to each user information in the user category, or may correspond to each item in the item category. The embedding may be used for natural language processing, recommendation systems, advertising, speech recognition, etc., but the inventive concept is not limited thereto.

[0047] In an exemplary embodiment, the memory modules 110 to 1n0 may perform tensor operations (or tensor operations) based on the stored embeddings. The memory modules 110 to 1n0 may generate new embeddings (hereinafter referred to as "reduced embeddings") through tensor operations. In this case, the tensor operations may be reduction operations including addition operations, subtraction operations, multiplication operations, juxtaposition operations, and average operations. For example, the memory modules 110 to 1n0 may generate reduced embeddings by performing tensor operations based on the first embedding and the second embedding. In this case, the data size of the reduced embedding may be the same as the data size of each of the first embedding and the second embedding, but may be smaller than the total data size of the first embedding and the second embedding. That is, the memory modules 110 to 1n0 may generate reduced embeddings by preprocessing the stored embeddings.

[0048] In an exemplary embodiment, the main processor 200 may receive the reduced embedding from the memory modules 110 to 1n0 through the first bus 1001 and may process the reduced embedding based on a neural network. That is, the main processor 200 may directly perform inference and training operations based on the reduced embedding without using the dedicated processor 300.

[0049] In another embodiment, the main processor 200 may receive the reduced embedding from the memory modules 110 to 1n0, and may transmit the reduced embedding to the special-purpose processor 300 through the second bus 1002. In this case, the special-purpose processor 300 may process the reduced embedding based on the neural network. That is, the special-purpose processor 300 may perform inference and training operations by using the reduced embedding. However, the inventive concept is not limited thereto, and the inference and training operations may be performed by both the main processor 200 and the special-purpose processor 300.

[0050] As described above, the neural network acceleration system 1000 may perform inference and training operations based on embedding. In this case, the memory modules 110 to 1n0 may preprocess the stored embedding without using the main processor 200 and the special processor 300, and may generate a reduced embedding by preprocessing. Therefore, at least one of the main processor 200 and the special processor 300 may receive the reduced embedding from the memory modules 110 to 1n0, and may perform inference and training operations based on the received reduced embedding. The reduced embedding may be transmitted to the main processor 200 and the special processor 300 via the first bus 1001 and the second bus 1002.

[0051] When the embeddings stored by the memory modules 110 to 1n0 are not preprocessed, the non-preprocessed embeddings may be transmitted to the main processor 200 and the special-purpose processor 300 through the first bus 1001 and the second bus 1002. Since each of the first bus 1001 and the second bus 1002 has a limited bandwidth and the data size of the embeddings is very large, the delay in transmitting the embeddings to the main processor 200 and the special-purpose processor 300 may be large. Therefore, the time required for inference and training operations may increase.

[0052] When the reduced embedding generated by preprocessing is transmitted to the main processor 200 and the dedicated processor 300 through the first bus 1001 and the second bus 1002, since the data size of the reduced embedding is relatively small compared to the embedding that is not preprocessed, the reduced embedding can be transmitted faster than the embedding (i.e., the latency is reduced). Therefore, the time required for inference and training operations can be reduced. That is, the neural network acceleration system 1000 can quickly perform inference and training operations by reducing the data size of the embedding transmitted under a limited bandwidth.

[0053] Figure 2 is a block diagram showing a neural network acceleration system according to another embodiment of the present invention. Figure 2 , the neural network acceleration system 2000 includes a first memory module 410 to an nth memory module 4n0, a main processor 500, a dedicated processor 600, a first bus 2001 and a second bus 2002. Figure 1 The components of the neural network acceleration system 1000 operate similarly, so additional description may be omitted to avoid redundancy.

[0054] The main processor 500 may perform inference and training operations by controlling the special processor 600. The special processor 600 may perform inference and training operations based on various neural network algorithms under the control of the main processor 500. The special processor 600 may perform inference and training operations based on data provided from the memory modules 410 to 4n0. The memory modules 410 to 4n0 may store data in an internal memory or output data stored in the internal memory under the control of the main processor 500 or the special processor 600.

[0055] The main processor 500 can communicate with the dedicated processor 600 through the first bus 2001, and the dedicated processor 600 can communicate with the memory module 410 to the memory module 4n0 through the second bus 2002. For example, the first bus 2001 can be based on one of various standards such as PCIe, AXI, AMBA, NVLink, etc. The second bus 2002 can be based on an interface protocol with a bandwidth equal to or greater than the bandwidth of the first bus 2001. For example, the second bus 2002 can be based on a common application programming interface (CAPI), Gen-Z, a cache consistency interconnect for accelerators (CCIX), a computing fast link (CXL), NVLink, and one of BlueLink. However, the inventive concept is not limited thereto, and the second bus 2002 can be based on one of various standards such as PCIe, NVMe, AXI, AMBA, etc.

[0056] In an exemplary embodiment, the memory modules 410 to 4n0 may store embeddings. The memory modules 410 to 4n0 may perform tensor operations based on the stored embeddings. The memory modules 410 to 4n0 may generate reduced embeddings through tensor operations. That is, the memory modules 410 to 4n0 may generate reduced embeddings by preprocessing the embeddings.

[0057] In an exemplary embodiment, the special purpose processor 600 may receive the reduced embedding from the memory modules 410 to 4n0 through the second bus 2002 and may process the reduced embedding based on the neural network. That is, the special purpose processor 600 may perform inference and training operations by using the reduced embedding.

[0058] As described above, the neural network acceleration system 2000 can perform inference and training operations based on the embedding. In this case, the dedicated processor 600 can directly receive the reduced embedding from the memory modules 410 to 4n0 through the second bus 2002. That is, the dedicated processor 600 can receive the reduced embedding without passing through the first bus 2001, and can perform inference and training operations using the reduced embedding. Therefore, the neural network acceleration system 2000 can be compared to Figure 1 The neural network acceleration system 1000 performs inference and training operations faster.

[0059] In the following, for the sake of convenience, the Figure 1 The neural network acceleration system 1000 describes in detail the operation of the neural network acceleration system according to an embodiment of the inventive concept.

[0060] Figure 3 is a flowchart describing the operation of a neural network acceleration system according to an embodiment of the inventive concept. Figure 1 and Figure 3In operation S1100, the neural network acceleration system 1000 may store the embedding segments generated by splitting the embedding in the memory modules 110 to 1n0. For example, the main processor 200 may generate the embedding segments by splitting the embedding based on a preset standard. The main processor 200 may allocate the embedding segments to the memory modules 110 to 1n0 so that the generated embedding segments are distributed and stored in the memory modules 110 to 1n0.

[0061] In operation S1200, the neural network acceleration system 1000 may collect embeddings (i.e., embedding lookup). For example, in inference and training operations, each of the memory modules 110 to 1n0 may collect at least one of the stored embedding segments without using the main processor 200. In this case, the collected embedding segments may be stored in a storage space corresponding to a continuous address among the storage spaces of each of the memory modules 110 to 1n0 (hereinafter, referred to as a continuous address space). That is, in the embedding lookup operation, the embedding segment may not be transmitted to the main processor 200.

[0062] In operation S1300, the neural network acceleration system 1000 may generate a reduced embedding by processing the collected embedding segments using tensor operations. For example, in inference and training operations, each of the memory modules 110 to 1n0 may perform tensor operations on the collected embedding segments. Thus, the memory modules 110 to 1n0 may generate a reduced embedding. The memory modules 110 to 1n0 may send the reduced embedding to the main processor 200 through the first bus 1001.

[0063] In operation S1400, the neural network acceleration system 1000 may process the reduced embedding based on the neural network. As an example, the main processor 200 may process the reduced embedding sent from the memory modules 110 to 1n0 based on the neural network. As another example, the main processor 200 may transmit the reduced embedding to the special-purpose processor 300 through the second bus 1002. The special-purpose processor 300 may process the reduced embedding sent from the main processor 200 based on the neural network.

[0064] Figure 4 2 is a diagram showing an embedded segment according to an embodiment of the present inventive concept. Figure 4 Detailed Description Figure 3 The operation of operation S1100.

[0065] Reference Figure 1 and Figure 4, the main processor 200 may split each of the first embedded EBD1 to the kth embedded EBDk (k is a natural number greater than or equal to 1). The first embedded EBD1 to the kth embedded EBDk may correspond to objects of the same category, respectively. For example, the first embedded EBD1 may correspond to the first user in the user category, and the second embedded EBD2 may correspond to the second user in the user category.

[0066] The main processor 200 may generate an embedded segment by splitting each of the embedded EBD1 to EBDk. For example, the main processor 200 may split the first embedded EBD1 to generate embedded segments SEG11 to SEG1n. Specifically, the main processor 200 may split the first embedded EBD1 into "n" pieces according to the number of memory modules 110 to 1n0. The main processor 200 may split the first embedded EBD1 so that each of the embedded segments SEG11 to SEG1n has the same size. However, the inventive concept is not limited thereto, and the main processor 200 may split the embedded according to various standards.

[0067] The main processor 200 may store the embedded segments in the memory modules 110 to 1n0. The main processor 200 may allocate the embedded segments in the memory modules 110 to 1n0 so that the embedded segments are distributed and stored in the memory modules 110 to 1n0. For example, the main processor 200 may store the embedded segment groups ESG1 to ESGn in the memory modules 110 to 1n0, respectively. In this case, the first memory module 110 may store the first embedded segment group ESG1. The first embedded segment group ESG1 may include embedded segments SEG11 and SEG21 to SEGk1. The embedded segments SEG11 and SEG21 to SEGk1 may correspond to the first embedded EBD1 to the kth embedded EBDk, respectively. The second memory module 120 may store the second embedded segment group ESG2. The second embedded segment group ESG2 may include embedded segments SEG12 and SEG22 to SEGk2. The embedded segments SEG12 and SEG22 to SEGk2 may correspond to the first embedded EBD1 to the kth embedded EBDk, respectively. That is, each of the memory modules 110 to 1n0 may store an embedding segment group constituting embeddings EBD1 to EBDk.

[0068] Figure 5 is a diagram illustrating the embedding of reduction according to an embodiment of the inventive concept. Figure 5 Detailed Description Figure 3 Operations S1200 and S1300. Figure 1 , Figure 4 and Figure 5 , each of the first memory module 110 to the nth memory module 1n0 may store a corresponding embedded segment group, such as Figure 4For example, the first memory module 110 may store embedded segments SEG11 to SEGk1.

[0069] The memory modules 110 to 1n0 may collect at least one of the stored embedding segments. In an exemplary embodiment, the memory modules 110 to 1n0 may collect embedding segments corresponding to the embedding selected by the main processor 200. For example, the first memory module 110 may collect embedding segments SEG11 to SEGk1 corresponding to the first embedding EBD1 to the kth embedding EBDk. The second memory module 120 may collect embedding segments SEG12 to SEGk2 corresponding to the first embedding EBD1 to the kth embedding EBDk.

[0070] The memory modules 110 to 1n0 may generate reduced embedded REBD by tensor operations on the collected embedded segments. In this case, each of the memory modules 110 to 1n0 may generate one of the segments RES1 to RESn of the reduced embedded REBD. For example, the first memory module 110 may generate the reduced embedded segment RES1 based on the embedded segments SEG11 to SEGk1. The second memory module 120 may generate the reduced embedded segment RES2 based on the embedded segments SEG12 to SEGk2.

[0071] As described above, each of the memory modules 110 to 1n0 may collect embedding segments, and may generate reduced embedding segments by tensor operations on the collected embedding segments. In this case, the reduced embedding segments generated from the memory modules 110 to 1n0 may form a reduced embedding REBD. The size of the reduced embedding REBD may be smaller than the embedding selected by the main processor 200 or the total embedding stored in the memory modules 110 to 1n0. Therefore, in inference and training operations, when the reduced embedding REBD generated from the memory modules 110 to 1n0 is transmitted to the main processor 200, latency may be reduced under limited bandwidth.

[0072] In the following, reference will be made to FIG. 6A to FIG. 7 A memory module according to an embodiment of the inventive concept is described in detail.

[0073] FIG. 6A to FIG. 6D 1 is a block diagram showing a memory module according to an embodiment of the present invention. FIG. 6A to FIG. 6D An example in which the memory module 700 generates the reduced embedding segment RES is described. FIG. 6A to FIG. 6D The memory module 700 may correspond to Figure 1Each of the memory modules 110 to 1n0. The memory module 700 may include a buffer device 710 and first to mth memory devices 721 to 72m. The buffer device 710 and the memory devices 721 to 72m may be implemented using different semiconductor packages and may be respectively disposed on one printed circuit board.

[0074] The buffer device 710 may control the operation of the memory devices 721 to 72m. The buffer device 710 may respond to a request from an external host device (eg, Figure 1 The memory devices 721 to 72m are controlled by commands sent by the main processor 200.

[0075] Each of the memory devices 721 to 72m may output data from an internal memory unit or may store data in an internal memory unit under the control of the buffer device 710. For example, each of the memory devices 721 to 72m may be implemented as a volatile memory device such as SRAM and DRAM or a nonvolatile memory device such as a flash memory, PRAM, MRAM, RRAM, and FRAM. For example, each of the memory devices 721 to 72m may be implemented as one chip or package.

[0076] exist FIG. 6A to FIG. 6D , the memory devices 721 to 72m are illustrated as “m” memory devices, but the memory module 700 may include at least one or more memory devices.

[0077] Reference Fig. 6A , the buffer device 710 may receive the embedded segment group ESG. For example, the embedded segment group ESG may include the first to kth embedded segments SEG1 to SEGp to SEGk (where "p" is a natural number "k" or less). For example, as shown in FIG. Figure 4 As described, the buffer device 710 may receive at least one of the embedded segment groups ESG1 to ESGn generated from embedding EBD1 to EBDk. The buffer device 710 may store the embedded segment group ESG in at least one of the memory devices 721 to 72m. For example, the buffer device 710 may store the first embedded segment SEG1 in the first memory device 721, may store the pth embedded segment SEGp in the second memory device 722, and may store the kth embedded segment SEGk in the mth memory device 72m. The buffer device 710 may store the embedded segment group ESG so that the embedded segment group ESG is distributed in the memory devices 721 to 72m, but the inventive concept is not limited thereto. For example, the buffer device 710 may store the first embedded segment SEG1 to the kth embedded segment SEGk in the first memory device 721.

[0078] In another embodiment, the buffer device 710 may split each of the embedded segments SEG1 to SEGk into a plurality of fragments, and may store the fragments in the memory devices 721 to 72m. For example, the buffer device 710 may generate the first fragment to the mth fragment by splitting the embedded segment SEG1 according to the number (i.e., m) of the memory devices 721 to 72m. In this case, the buffer device 710 may store the first fragment in the first memory device 721, and may store the second fragment in the second memory device 722. Similarly, the buffer device 710 may store the remaining fragments in the corresponding memory devices. Therefore, when the buffer device 710 reads each of the embedded segments SEG1 to SEGk from the memory devices 721 to 72m or writes each of the embedded segments SEG1 to SEGk in the memory devices 721 to 72m, the buffer device 710 may maximize the use of the bus bandwidth between the buffer device 710 and the memory devices 721 to 72m.

[0079] Reference Figure 6B In inference and training operations, the buffer device 710 may respond to requests from an external host device (e.g., Figure 1 The embedded segments SEG1 to SEGp stored in the memory devices 721 to 72m are outputted according to the embedded search instruction of the host processor 200. For example, the outputted embedded segments SEG1 to SEGp may correspond to the embedded segments SEG1 to SEGp selected by the host device.

[0080] Reference Figure 6C , the buffer device 710 may store the output embedded segments SEG1 to SEGp in a continuous address space among the storage spaces of the memory devices 721 to 72m. In an exemplary embodiment, the buffer device 710 may store the embedded segments SEG1 to SEGp in one of the memory devices 721 to 72m. For example, the buffer device 710 may store the embedded segments SEG1 to SEGp in a continuous address space of the first memory device 721. Therefore, the embedded segments SEG1 to SEGp may be collected. That is, an embedded search operation may be performed.

[0081] Reference Fig.6D , the buffer device 710 may perform tensor operations based on the collected embedding segments SEG1 to SEGp. The buffer device 710 may process the embedding segments SEG1 to SEGp through tensor operations. For example, the buffer device 710 may perform tensor operations such as addition operations, subtraction operations, multiplication operations, concatenation operations, and average operations on the embedding segments SEG1 to SEGp. Therefore, the buffer device 710 may generate a reduced embedding segment RES. The buffer device 710 may output the generated reduced embedding segment RES. For example, the buffer device 710 may send the reduced embedding segment RES to Figure 1 The main processor 200.

[0082] For example, with FIG. 6A to FIG. 6D Unlike what is described above, the embedded segments SEG1 to SEGp are sent to Figure 1 The embedded segments SEG1 to SEGp transmitted to the main processor 200 are transmitted back to the memory module 700 and can then be stored in a continuous address space of the memory module 700. In this way, when the embedded segments are transmitted between each of the plurality of memory modules and the main processor 200, the latency of the embedded lookup may be increased due to limited bandwidth.

[0083] In contrast, as described above, the memory module 700 may collect the embedding segments SEG1 to SEGp without transmitting the embedding segments SEG1 to SEGp to the outside. Therefore, the memory module 700 may collect the embedding segments SEG1 to SEGp regardless of the limited bandwidth. That is, even if the number of memory modules increases, each of the memory modules may perform embedding lookup without being limited by bandwidth. Therefore, as the number of memory modules increases, the available memory bandwidth of the neural network acceleration system according to an embodiment of the inventive concept may increase in proportion to the number of memory modules.

[0084] Although the description is based on the embedded segments SEG1 to SEGp collected by the memory module 700 through FIG. 6A to FIG. 6D The reduced embedding segments RES are generated by performing tensor operations in , but the inventive concept is not limited thereto. For example, the memory module 700 may send the collected embedding segments SEG1 to SEGp to the main processor 200 without performing tensor operations separately.

[0085] Figure 7 It is shown FIG. 6A to FIG. 6D A block diagram of an example of a buffer device. FIG. 6A to FIG. 7 , the buffer device 710 may include a device controller 711 and a tensor operator 713. The device controller 711 may control the operation of the buffer device 710 and the memory devices 721 to 72m. For example, the device controller 711 may control the operation of the tensor operator 713.

[0086] The tensor operator 713 may perform tensor operations under the control of the device controller 711. For example, the tensor operator 713 may be implemented as an arithmetic logic unit that performs addition operations, subtraction operations, multiplication operations, concatenation operations, and average operations. The tensor operator 713 may provide result data calculated by the tensor operation to the device controller 711.

[0087] The device controller 711 may include a buffer memory 712. The device controller 711 may store data provided from the outside or data generated therein in the buffer memory 712. The device controller 711 may output the data stored in the buffer memory 712 to the outside of the buffer device 710. Figure 7 The buffer memory 712 in FIG. 7 is shown as being located inside the device controller 711 , but the inventive concept is not limited thereto. For example, the buffer memory 712 may be located outside the device controller 711 .

[0088] The device controller 711 may control the embedding segments SEG1 to SEGp to be output from the memory devices 721 to 72m. For example, the device controller 711 may control the embedding segments SEG1 to SEGp collected in one of the memory devices 721 to 72m to be output. The device controller 711 may store the outputted embedding segments SEG1 to SEGp in the buffer memory 712. The device controller 711 may provide the embedding segments SEG1 to SEGp stored in the buffer memory 712 to the tensor operator 713.

[0089] The tensor operator 713 may perform a tensor operation based on the embedding segments SEG1 to SEGp, and may generate a reduced embedding segment RES. The tensor operator 713 may send the generated reduced embedding segment RES to the device controller 711. The device controller 711 may store the reduced embedding segment RES in the buffer memory 712. The device controller 711 may output the reduced embedding segment RES stored in the buffer memory 712. For example, the device controller 711 may send the reduced embedding segment RES to the device controller 711. Figure 1 The main processor 200.

[0090] As described above, the memory module 700 according to an embodiment of the inventive concept can generate a reduced embedding segment RES by performing a tensor operation on the embedding segments SEG1 to SEGp. In this case, the data size of the reduced embedding segment RES can be smaller than the data size of the entire embedding segments SEG1 to SEGp. Therefore, when the reduced embedding segment RES is sent to the main processor 200 through the first bus 1001 or is sent to the dedicated processor 300 through the first bus 1001 and the second bus 1002, the delay can be reduced under limited bandwidth. Therefore, the main processor 200 or the dedicated processor 300 can quickly perform inference and training operations based on the reduced embedding segment RES (i.e., the reduced embedded REBD).

[0091] Figure 8 is a block diagram showing an extended example of a neural network acceleration system according to an embodiment of the inventive concept. Figure 8, the neural network acceleration system 3000 may include a central processing unit 3100, a memory 3200, a neural processing unit 3300, a user interface 3400, a network interface 3500, and a bus 3600. For example, the neural network acceleration system 3000 may be implemented as one of a desktop, a laptop computer, an embedded system, a server, a car, a mobile device, and an artificial intelligence system.

[0092] The central processing unit 3100 may control the neural network acceleration system 3000. For example, the central processing unit 3100 may control the operation of the memory 3200, the neural processing unit 3300, the user interface 3400, and the network interface 3500. The central processing unit 3100 may send data and commands to the components of the neural network acceleration system 3000 through the bus 3600, and may receive data from the components. For example, the central processing unit 3100 may use a reference Figure 1 and Figure 2 The embodiment is implemented by one of the main processors 200 and 500 described above.

[0093] The memory 3200 may store data or may output stored data. The memory 3200 may store data to be processed or data processed by the central processing unit 3100 and the neural processing unit 3300. For example, the memory 3200 may include a reference FIG. 6A to FIG. 6D The memory 3200 may store the embedding segments and generate reduced embeddings by performing tensor operations on the embedding segments. In inference and training operations, the memory 3200 may transmit the reduced embeddings to the central processing unit 3100 or the neural processing unit 3300.

[0094] The neural processing unit 3300 may perform inference and training operations based on various neural network algorithms under the control of the central processing unit 3100. For example, the neural processing unit 3300 may use reference Figure 1 and Figure 2 The neural processing unit 3300 may be implemented by one of the dedicated processors 300 and 600 described above. The neural processing unit 3300 may perform inference and training operations based on the reduced embedding transmitted from the memory 3200. The neural processing unit 3300 may transmit the results of the inference and training to the central processing unit 3100. Therefore, the central processing unit 3100 may output the results of the inference and training through the user interface 3400 or may control the neural network acceleration system 3000 according to the results of the inference and training.

[0095] Describes Figure 8The neural processing unit 3300 in the system performs inference and training operations, but the inventive concept is not limited thereto. For example, the neural network acceleration system 3000 may include a graphics processing device instead of the neural processing unit 3300. In this case, the graphics processing device may perform inference and training operations based on the neural network.

[0096] The user interface 3400 may be configured to exchange information with a user. The user interface 3400 may include a user input device that receives information from a user, such as a keyboard, a mouse, a touch panel, a motion sensor, a microphone, etc. The user interface 3400 may include a user output device that provides information to a user, such as a display device, a speaker, a beam projector, a printer, etc. For example, the neural network acceleration system 3000 may start inference and training operations through the user interface 3400, and may output the results of inference and training.

[0097] The network interface 3500 may be configured to exchange data with an external device wirelessly or wired. For example, the neural network acceleration system 3000 may receive embedding learned from an external device through the network interface 3500. The neural network acceleration system 3000 may send inference and training results to an external device through the network interface 3500.

[0098] Bus 3600 may transmit commands and data between components of neural network acceleration system 3000. For example, bus 3600 may include reference Figure 1 and Figure 2 Buses 1001, 1002, 2001 and 2002 are described.

[0099] According to an embodiment of the inventive concept, a neural network acceleration system may include a memory module capable of preprocessing stored embeddings. The memory module may reduce the data size of the embedding to be sent to the processor through preprocessing. The memory module may send the embedding generated by the preprocessing to the processor, and the processor may perform inference and training operations based on the embedding. In this way, since the embedding is preprocessed by the memory module, the size of the data sent from the memory module to the processor may be reduced. Therefore, the delay depending on the embedding transmission between the memory module and the processor may be reduced, and the neural network acceleration system may quickly perform inference and training operations.

[0100] The above-described contents are specific embodiments for realizing the inventive concept. The inventive concept may include not only the above-described embodiments, but also embodiments in which the design can be simply or easily changed. In addition, the inventive concept may also include technologies that are easily changed to be realized using the embodiments. Therefore, the scope of the inventive concept is not limited to the described embodiments, but should be defined by the claims and their equivalents.

Claims

1. A neural network acceleration system, comprising: A first memory module is configured to generate a first reduced embedding segment through a tensor operation based on the first embedding first segment and the second embedding second segment; a second memory module configured to generate a second reduced embedding segment through the tensor operation based on the first embedded third segment and the second embedded fourth segment; a main processor configured to receive the first reduced embedded segment and the second reduced embedded segment via a first bus; as well as a dedicated processor configured to process a reduced embedding based on a neural network algorithm, the reduced embedding comprising the first reduced embedding segment and the second reduced embedding segment transmitted from the main processor via a second bus, The data size of the first reduced embedded segment is smaller than the total data size of the first segment of the first embedding and the second segment of the second embedding, and the data size of the second reduced embedded segment is smaller than the total data size of the third segment of the first embedding and the fourth segment of the second embedding.

2. The neural network acceleration system according to claim 1, wherein: The first embedding corresponds to a first object of a particular class, and wherein the second embedding corresponds to a second object of the particular class.

3. The neural network acceleration system according to claim 1, wherein: The first memory module comprises: at least one memory device configured to store the first segment and the second segment; and A tensor operator is configured to perform the tensor operation based on the first segment and the second segment.

4. The neural network acceleration system according to claim 1, wherein: The first bus is configured to transfer the first reduced embedded segment and the second reduced embedded segment from the first memory module and the second memory module to the main processor based on a first bandwidth, respectively, and The second bus is configured to transmit the first reduced embedded segment and the second reduced embedded segment from the main processor to the dedicated processor based on a second bandwidth.

5. The neural network acceleration system according to claim 1, wherein: The main processor is also configured to store the first segment generated by splitting the first embedding and the second segment generated by splitting the second embedding in the first memory module, and is also configured to store the third segment generated by splitting the first embedding and the fourth segment generated by splitting the second embedding in the second memory module.

6. The neural network acceleration system according to claim 5, wherein: The main processor is further configured to split the first embedding such that a data size of the first segment is the same as a data size of the third segment, and the main processor is further configured to split the second embedding such that a data size of the second segment is the same as a data size of the fourth segment.

7. The neural network acceleration system according to claim 1, wherein: The dedicated processor includes at least one of a graphics processing device and a neural network processing device.

8. The neural network acceleration system according to claim 1, wherein: The first memory module is further configured to collect the first segment and the second segment in a storage space corresponding to consecutive addresses, and The first reduced embedding segment is generated based on the collected first segment and the second segment.

9. The neural network acceleration system according to claim 3, wherein: The at least one memory device is implemented as a dynamic random access memory.

10. The neural network acceleration system according to claim 1, wherein: The tensor operation includes at least one of an addition operation, a subtraction operation, a multiplication operation, a concatenation operation, and an average operation.

11. A method of operating a neural network acceleration system, the neural network acceleration system comprising a first memory module, a second memory module, a main processor and a dedicated processor, the method comprising: storing, by the main processor, a first segment generated by splitting the first embedding and a second segment generated by splitting the second embedding in the first memory module, and storing, by the main processor, a third segment generated by splitting the first embedding and a fourth segment generated by splitting the second embedding in the second memory module; The first memory module generates a first reduced embedding segment through tensor operation based on the first segment and the second segment, and the second memory module generates a second reduced embedding segment through the tensor operation based on the third segment and the fourth segment; Receiving, by the main processor through a first bus, the embedded segment of the first reduction and the embedded segment of the second reduction; as well as processing, by the dedicated processor, a reduced embedding based on a neural network algorithm, the reduced embedding comprising the first reduced embedding segment and the second reduced embedding segment transmitted from the main processor via a second bus, The data size of the first reduced embedded segment is smaller than the total data size of the first segment of the first embedding and the second segment of the second embedding, and the data size of the second reduced embedded segment is smaller than the total data size of the third segment of the first embedding and the fourth segment of the second embedding.

12. The method according to claim 11, wherein: The first embedding corresponds to a first object of a particular class, and wherein the second embedding corresponds to a second object of the particular class.

13. The method according to claim 11, wherein: The step of generating the first reduced embedding segment by the first memory module comprises: collecting, by the first memory module, the first segment and the second segment in a storage space corresponding to consecutive addresses; and The first reduced embedding segment is generated based on the collected first segment and the second segment.

14. The method according to claim 11, wherein: The first memory module includes at least one memory device and a tensor operator, the at least one memory device is used to store the first segment and the second segment, and the tensor operator is used to perform the tensor operation based on the first segment and the second segment.

15. The method according to claim 11, wherein: The first bus is used to transfer the first reduced embedded segment and the second reduced embedded segment from the first memory module and the second memory module to the main processor based on a first bandwidth, and the second bus is used to transfer the first reduced embedded segment and the second reduced embedded segment from the main processor to the dedicated processor based on a second bandwidth.

16. The method according to claim 11, wherein: The data size of the first segment is the same as the data size of the third segment, and the data size of the second segment is the same as the data size of the fourth segment.

17. The method according to claim 11, wherein: The dedicated processor includes at least one of a graphics processing device and a neural network processing device.

18. The method according to claim 14, wherein: The at least one memory device is implemented as a dynamic random access memory.

19. The method according to claim 11, wherein: The tensor operation includes at least one of an addition operation, a subtraction operation, a multiplication operation, a concatenation operation, and an average operation.

Images