Computing fast link storage device and computing device

By selecting appropriate computing circuits for computing fast link (CXL) storage devices and interfaces, the problems of GPU memory capacity and computing efficiency limitations are solved, and efficient LLM computing is achieved, reducing latency and power consumption.

CN119025018AInactive Publication Date: 2024-11-26SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410164364.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-08-21
Filing Date
2024-02-05
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Large Language Model (LLM) In computing systems with high computing performance requirements, due to the limitations of the memory capacity and computing efficiency of the GPU, the memory bandwidth and capacity are insufficient, affecting computing efficiency and power consumption.

Method used

Using computing fast link (CXL) storage devices, select appropriate computing circuits through the CXL interface to perform computing tasks, support high memory bandwidth and large memory capacity, optimize matrix-vector multiplication and matrix-matrix multiplication calculations, reduce latency and power consumption .

Benefits of technology

It realizes efficient calculation in the LLM computing system, shortens data movement time, reduces execution time and power consumption, and improves computing performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025018A_ABST
    Figure CN119025018A_ABST
Patent Text Reader

Abstract

A compute fast link (CXL) storage device includes: a storage device storing data; and a controller configured to read data from the storage device based on a first command received through a first protocol, select a calculation engine based on a second command received through a second protocol different from the first protocol, and control the calculation engine to perform calculation on the read data.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of Korean Patent Application No. 10-2023-0066421 filed on May 23, 2023, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2023-0109241 filed on August 21, 2023, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to storage devices, and more particularly, to a compute express link (CXL) storage device and a computing system including the same. Background Art

[0004] With the advent of artificial intelligence (AI) technology, large language models (LLMs) may become increasingly popular because LLMs can be used to provide services in various AI-based fields. For example, LLM-based services may tend to be larger model sizes that can provide higher performance services. As a result, LLMs may need to be implemented on computing systems that include large-capacity and / or high-bandwidth storage devices that may need to support high computing performance requirements. Possible methods for obtaining large-capacity memory and / or high computing performance may include the use of multiple graphics processing units (GPUs).

[0005] However, due to the structural characteristics of GPUs, the use of GPUs may be limited by memory capacity limitations and / or computational inefficiencies. Summary of the invention

[0006] One or more example embodiments of the present disclosure provide a high-capacity Compute Express Link (CXL) device, and a computing system that provides services based on a Large Language Model (LLM).

[0007] Furthermore, one or more example embodiments of the present disclosure provide a CXL device and a computing system that efficiently performs calculations by selecting a computing circuit based on the type of calculation to be performed.

[0008] According to one aspect of the present disclosure, a CXL storage device includes: a storage device storing data; and a controller configured to read data from the storage device based on a first command received through a first protocol, select a computing engine based on a second command received through a second protocol different from the first protocol, and control the computing engine to perform computing on the read data.

[0009] According to one aspect of the present disclosure, a computing system includes: a host configured to output at least one of a first computing command and a second computing command; an accelerator configured to operate based on the first computing command; a storage device configured to operate based on the second computing command; and a CXL interface configured to send the first computing command to the accelerator and send the second computing command to the storage device.

[0010] According to one aspect of the present disclosure, a CXL storage device includes: a computing engine, including a first computing circuit and a second computing circuit, the first computing circuit being configured to perform a first computing on input data and LLM data, the second computing circuit being configured to perform a second computing on the input data and the LLM data, the first computing being different from the second computing; and a scheduler, being configured to receive a computing command through a CXL interface, and select at least one of the first computing circuit and the second computing circuit based on the computing command.

[0011] Additional aspects may be set forth in part in the description which follows and, in part, may be obvious from the description, and / or may be learned by practice of the presented embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a block diagram of a computing system applying a compute express link (CXL) memory according to an embodiment;

[0014] Figure 2 is a schematic block diagram of a CXL memory according to an embodiment;

[0015] Figure 3 is a diagram of a matrix-vector multiplication computation performed by a computation circuit of a CXL memory according to an embodiment;

[0016] Figure 4 is a circuit diagram of calculation of a calculation circuit according to an embodiment;

[0017] Figure 5 is a block diagram of a CXL memory according to an embodiment;

[0018] Figure 6 is a schematic block diagram of a CXL memory according to an embodiment;

[0019] Figure 7 is a diagram of a matrix-matrix multiplication computation performed by a computation circuit of a CXL memory according to an embodiment;

[0020] Figure 8is a diagram of calculations of a calculation circuit according to an embodiment;

[0021] Fig. 9 is a circuit diagram of a processing element included in a computing circuit according to an embodiment;

[0022] Fig.10 is a block diagram of a CXL memory according to an embodiment;

[0023] Fig.11 is a block diagram of a processing engine according to an embodiment;

[0024] Fig.12 is a block diagram of a computing system using CXL memory according to an embodiment;

[0025] Fig.13 is a block diagram of a computing system using CXL memory according to an embodiment;

[0026] Fig.14 is a flowchart of a calculation method of a CXL memory according to an embodiment;

[0027] Fig.15 is a block diagram of a computing system for an application storage system according to an embodiment;

[0028] Fig.16 is a block diagram of a computing system according to an embodiment; and

[0029] Fig.17 is a block diagram of a server of an application computing system according to an embodiment. DETAILED DESCRIPTION

[0030] In the following detailed description, only certain embodiments of the present disclosure are shown and described by way of illustration. Those skilled in the art will appreciate that the described embodiments may be modified in various different ways without departing from the spirit and / or scope of the present disclosure.

[0031] Therefore, the drawings and descriptions should be considered illustrative in nature and not restrictive. Throughout the specification, similar reference numerals may represent similar elements. In the flowcharts described with reference to the drawings, the order of operations may be changed, several operations may be combined, some operations may be divided, and certain operations may not be performed.

[0032] In addition, unless explicit expressions such as "one" or "single" are used, expressions written in the singular may be interpreted as singular or plural. Terms including ordinal numbers such as first, second, etc. may be used to describe various components and may not be interpreted as limiting these components. These terms may be used for the purpose of distinguishing one constituent element from other constituent elements.

[0033] As used herein, each of such phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any possible combination of items listed together in corresponding ones of these phrases. It should be understood that if an element (e.g., a first element) is referred to as being "coupled", "coupled to", "connected to" or "connected to" another element (e.g., a second element) with or without the term "operably" or "communicatively", it means that the element can be coupled to the other element directly (e.g., wired), wirelessly, or via a third element.

[0034] Throughout this disclosure, references to "one embodiment," "an embodiment," "an example embodiment," or similar language may indicate that a particular feature, structure, or characteristic described in connection with the indicated embodiment is included in at least one embodiment of the present solution. Thus, throughout this disclosure, the phrases "in one embodiment," "in an embodiment," "in an example embodiment," and similar language may, but do not necessarily, all refer to the same embodiment.

[0035] The embodiments herein may be described and illustrated in terms of blocks that perform one or more functions described as shown in the accompanying drawings. These blocks (which may be referred to herein as units or modules, etc., or named as devices, logic, circuits, controllers, accelerators, counters, comparators, generators, converters, etc.) may be physically implemented by analog and / or digital circuits including one or more of logic gates, integrated circuits, microprocessors, microcontrollers, storage circuits, passive electronic components, active electronic components, optical components, etc.

[0036] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings.

[0037] Figure 1 is a block diagram of a computing system applying a compute express link (CXL) memory according to an embodiment.

[0038] refer to Figure 1 According to an embodiment, the computing system 100 may be and / or may be included in a user device, such as but not limited to a personal computer (PC), a laptop computer, a server, a data center, a media player, a digital camera, and / or a vehicle device (such as but not limited to a navigation system, a black box, a vehicle electrical device, etc.). Alternatively or additionally, the computing system 100 may be and / or may include a mobile system, such as but not limited to a portable communication terminal (such as a mobile phone), a smart phone, a tablet PC, a wearable device, a healthcare device, an Internet of Things (IoT) device, etc.

[0039] According to an embodiment, computing system 100 may include host 110 and CXL memory 130. Host 110 and CXL memory 130 may be connected and communicate through CXL interface 120. CXL interface 120 may include (e.g., implement) a lower layer protocol, such as but not limited to CXL.io protocol, CXL.mem protocol, CXL.cache protocol, etc.

[0040] The host 110 may control the overall operation of the computing system 100. In an embodiment, the host 110 may be and / or may include one of various processors, such as but not limited to a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a data processing unit (DPU), etc. In an embodiment, the host 110 may include a single-core processor and / or a multi-core processor.

[0041] The CXL memory 130 may include a CXL controller (CTRL) 131 and a plurality of memory devices (MEM) 132. The CXL controller 131 may be configured to store data in the plurality of memory devices 132. Alternatively or additionally, the CXL controller 131 may be configured to transmit the data stored in the plurality of memory devices 132 to the host 110 under the control of the host 110.

[0042] The plurality of storage devices 132 may include a first storage device 1321 to an nth storage device 132n, where n is a positive integer greater than zero (0). In an embodiment, the plurality of storage devices 132 may be configured to store data of a large language model (LLM). The CXL controller 131 may load data from the plurality of storage devices 132 according to an instruction of the host 110 and perform calculations on the data. In an embodiment, the plurality of storage devices 132 may be and / or may include a dynamic random access memory (DRAM). However, the present disclosure is not limited thereto. That is, the plurality of storage devices 132 may be and / or may include other types of storage devices.

[0043] In an embodiment, the host 110 and the CXL memory 130 may be configured to share the same interface. For example, the host 110 and the CXL memory 130 may communicate with each other via the CXL interface 120. In an embodiment, the CXL interface 120 may refer to a low-latency and high-bandwidth link that enables various connections between accelerators, storage devices, and / or various electronic devices. The CXL interface 120 may support dynamic protocol multiplexing memory access and input / output protocols, and may maintain storage consistency between storage spaces of interconnected devices.

[0044] In the following, for better understanding and convenience of description, it may be assumed that the host 110 and the CXL memory 130 communicate with each other through the CXL interface 120. However, the present disclosure is not limited thereto. That is, the host 110 and the CXL memory 130 may communicate with each other based on various other computing interfaces, such as but not limited to the Gen-Z protocol, the NVLink protocol, the Cache Coherent Interconnect for Accelerators (CCIX) protocol, the Open Coherent Accelerator Processor Interface (CAPI) protocol, etc.

[0045] In an embodiment, the host 110 may instruct the CXL memory 130 to perform a matrix-vector multiplication calculation. The host 110 may instruct the CXL controller 131 to perform the matrix-vector multiplication calculation by using the CXL.io protocol and / or the CXL.mem protocol of the CXL interface 120. For example, the host 110 may send a command instructing the matrix-vector multiplication calculation to the CXL controller 131 by using the CXL.io protocol, and load data as a calculation object of the plurality of storage devices 132 through the CXL controller 131 by using the CXL.mem protocol. For example, the data to be calculated may be a weight value (and / or filter data) of an LLM. The CXL controller 131 may receive input data from the host 110, and perform a matrix-vector multiplication calculation on the input data and the weight value. The CXL controller 131 may include a calculation circuit that performs the matrix-vector multiplication calculation. The calculation circuit may include an adder tree in which a plurality of multipliers and a plurality of adders perform calculations in a tree shape. In the adder tree, at the first stage of input data, a plurality of multipliers may be provided, and at the second stage which is the next stage after the first stage, a plurality of adders may be provided until the final stage which outputs the calculation result. The CXL controller 131 may perform matrix-vector multiplication calculation on the input data and the data of the plurality of storage devices 132 by using a calculation circuit.

[0046] In an embodiment, the host 110 may instruct at least one of a matrix-matrix multiplication calculation and a matrix-vector multiplication calculation to the CXL memory 130. The host 110 may instruct the CXL controller 131 to perform the matrix-matrix multiplication calculation and / or the matrix-vector multiplication calculation by using the CXL.io protocol and the CXL.mem protocol of the CXL interface 120. For example, the host 110 may send a command instructing the matrix-matrix multiplication calculation and / or the matrix-vector multiplication calculation to the CXL controller 131 by using the CXL.io protocol, and may load data as a calculation object of the plurality of storage devices 132 through the CXL controller 131 by using the CXL.mem protocol. For example, the data as the calculation object may be a weight value of an LLM. The CXL controller 131 may receive input data from the host 110, and perform the matrix-matrix multiplication calculation and / or the matrix-vector multiplication calculation on the input data and the weight value. The CXL controller 131 may include a first calculation circuit that performs the matrix-matrix multiplication calculation and a second calculation circuit that performs the matrix-vector multiplication calculation. The first calculation circuit may include a processing element (PE) array in which a plurality of processing elements are arranged. In an embodiment, one PE may include a multiplier and an adder. However, the present disclosure is not limited thereto. That is, a PE may include more than one (1) multiplier and / or more than one (1) adder. The second calculation circuit may include an adder tree in which a plurality of multipliers and a plurality of adders perform calculations in a tree shape. A plurality of multipliers may be arranged at a first stage of the adder tree, and a plurality of adders may be arranged from a second stage to a final stage. The CXL controller 131 may select the first calculation circuit or the second calculation circuit based on a command of the host 110. For example, when the host 110 instructs a matrix-matrix multiplication calculation, the CXL controller 131 may select the first calculation circuit. As another example, when the host 110 instructs a matrix-vector multiplication calculation, the CXL controller 131 may select the second calculation circuit. The CXL controller 131 may perform a matrix-matrix multiplication calculation and / or a matrix-vector multiplication calculation on input data and data of the plurality of storage devices 132 by using the selected calculation circuit.

[0047] In the case where the data of LLMs grows exponentially, accelerators such as GPUs may support limited memory bandwidth and / or limited memory capacity. Therefore, it may be necessary to distribute and store LLMs on each accelerator chip. As a result, a large amount of inter-chip communication may be required to process LLMs, resulting in relatively large overhead in terms of execution time and power consumption.

[0048] According to an embodiment, the host 110 can serve the LLM by using the CXL memory 130, wherein the CXL memory 130 supports a relatively high memory bandwidth and a relatively large memory capacity by using the CXL interface 120. That is, according to an embodiment, the computing system 100 can apply the CXL memory 130 storing the data of the LLM to shorten the movement time of the data of the LLM and reduce the overhead in terms of execution time and power consumption. In addition, according to an embodiment, the computing system 100 includes the CXL memory 130 that selects an appropriate computing circuit according to the computing type of the LLM, and thus can perform efficient computing with relatively low latency and relatively low power consumption.

[0049] Figure 2 is a schematic block diagram of a CXL memory according to an embodiment. Figure 3 is a diagram of a matrix-vector multiplication computation performed by computation circuitry of a CXL memory, according to an embodiment. Figure 4 is a circuit diagram of calculation of a calculation circuit according to an embodiment.

[0050] refer to Figure 2 According to an embodiment, the CXL memory 200 may include a CXL controller (CTRL) 210 and a plurality of memory devices (MEM) 220. The plurality of memory devices 220 may include first to nth memory devices 220_1 to 220_n and may be configured to store LLM data. The CXL controller 210 may include a computation circuit (CORE) 211 that performs computations on LLM data retrieved from the plurality of memory devices 220.

[0051] CXL memory 200 may include and / or may be similar in many respects to the above-referenced Figure 1 The CXL memory 130 described above may include additional features not mentioned above. In addition, the CXL controller 210 and the plurality of storage devices 220 may each include and / or may be similar in many respects to the CXL memory 130 described above. Figure 1 The CXL controller 131 and the plurality of storage devices 132 described above may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference may be omitted. Figure 1 The description of CXL memory 200 , CXL controller 210 , and plurality of storage devices 220 is repeated.

[0052] According to an embodiment, CXL controller 210 may receive a signal from a host (e.g., Figure 1For example, CXL controller 210 may receive commands from the host via CXL.io protocol 121. CXL controller 210 may process the commands of the host by using computation circuit 211. For example, computation circuit 211 may process an instruction from the host indicating a matrix-vector multiplication calculation.

[0053] The computation circuit 211 may perform matrix-vector multiplication computations on input data represented as vectors and LLM data represented as matrices. In an embodiment, the input data and the LLM data may be stored in a plurality of storage devices 220 and moved to the computation circuit 211 for computation. In an embodiment, the input data may be input from the host to the computation circuit 211 via the CXL.mem protocol 122, and the LLM data may be stored in a plurality of storage devices 220 and moved to the computation circuit 211 for computation. The computation circuit 211 may perform operations as described in reference to Figure 3 Describes the matrix-vector multiplication calculation. Figure 2 and Figure 3 , the calculation circuit 211 can perform a multiplication calculation on the input data INPUT and the LLM data WEIGHT to generate the output data OUTPUT. The calculation circuit 211 may include an adder tree for performing a multiplication calculation (e.g., inner product calculation) on a vector of size 1×z (e.g., one row and z columns) of the input data INPUT and a matrix of size z×1 (e.g., z rows and one column), where z is a positive integer greater than zero (0). The calculation circuit 211 may include m adder trees for performing multiplication calculations in parallel, where m is a positive integer greater than zero (0). That is, the calculation circuit 211 can perform a multiplication calculation on a vector of size 1×z of the input data INPUT and a matrix of size z×m of the LLM data WEIGHT (e.g., z rows and m columns).

[0054] refer to Figure 4 , the computing circuit 211 can perform matrix-vector multiplication calculations in parallel. For example, the computing circuit 211 may include a first computing channel 211_1, a second computing channel 211_2 to an m-th computing channel 211_m, and each computing channel may include an adder tree. By performing a matrix-vector multiplication calculation in each of the first computing channel 211_1 to the m-th computing channel 211_m, the computing circuit 211 can implement the matrix-vector multiplication calculation in parallel. Figure 4 In FIG. 1 , the operation of the first computing channel 211_1 is described, and for the sake of brevity, repeated descriptions of the second computing channel 211_2 to the mth computing channel 211_m are omitted.

[0055] The first calculation channel 211_1 may include an adder tree. The adder tree may include a plurality of multipliers (e.g., a first multiplier 151_1, a second multiplier 151_2, a third multiplier 151_3, a fourth multiplier 151_4 to a P-th multiplier 151_P, where P is a positive integer greater than zero (0)) and a plurality of adders (e.g., a first top adder 152_1, a second top adder 152_2 to an R-th top adder 152_R, a first bottom adder 153_1, a second bottom adder 153_2, and a final adder 154, where R is a positive integer greater than zero (0)). The plurality of multipliers 151_1 to 151_P and the plurality of adders 152_1 to 154 may perform calculations using a tree shape. For example, the first calculation channel 211_1 may include multiple stages and / or layers (e.g., a first stage LV_1, a second stage LV_2, a (K-1)th stage LV_(K-1), and a Kth stage LV_K, where K is a positive integer greater than zero (0)). In the first stage LV_1, a plurality of multipliers 151_1 to 151_P may be provided. A plurality of adders 152_1 to 154 may be provided from the second stage LV_2, which is a next stage of the first stage LV_1, to the Kth stage LV_K which outputs the calculation result OPR.

[0056] In an embodiment, a plurality of (P) multipliers (e.g., 151_1 to 151_P) may be provided at the first stage LV_1, and P may be a power function of two (2) (e.g., P=2^X, where X is a positive integer greater than or equal to zero (0)). That is, the plurality of multipliers 151_1 to 151_P may receive P input data (e.g., first input data A1, second input data A2, third input data A3, fourth input data A4 to Pth input data AP) and P weight values ​​(e.g., first weight value W1, second weight value W2, third weight value W3, fourth weight value W4 to Pth weight value WP). The plurality of multipliers 151_1 to 151_P may perform multiplication calculations on the input data A1 to AP and the weight values ​​W1 to WP. The plurality of multipliers 151_1 to 151_P may output the results of the multiplication calculations performed on the input data A1 to AP and the weight values ​​W1 to WP to the adders 152_1 to 152_R of the second stage LV_2. For example, the calculation results of the first multiplier 151_1 and the second multiplier 151_2 may be input to the first top adder 152_1, the calculation results of the third multiplier 151_3 and the fourth multiplier 151_4 may be input to the second top adder 152_2, and the calculation results of the (P-1)th multiplier 151_(P-1)th multiplier 151_P and the Pth multiplier 151_P may be input to the Rth top adder 152_R.

[0057] In an embodiment, a plurality of (Q) adders (e.g., 152_1 to 154) may be provided from the second stage LV_2 to the Kth stage LV_K. For example, Q may be a positive integer that is 1 less than P (e.g., Q=P-1). That is, if P is a power function of two (2) (e.g., P=2^X), Q may be equal to 2^X-1. The number of top adders (e.g., the first top adder 152_1 to the Rth top adder 152_R) provided at the second stage LV_2 may be equal to half of P (e.g., R=P / 2). The number of adders provided at any stage from the second stage to the Kth stage (e.g., LV_2 to LV_K) may be half of the number of adders provided at the previous stage. In this manner, the (K-1)th stage LV_(K-1) may be composed of two adders (e.g., the first bottom adder 153_1 and the second bottom adder 153_2), and the Kth stage LV_K may be composed of one adder (e.g., the final adder 154). That is, the number of the plurality of adders 152_1 to 154 provided at the second stage LV_2 to the Kth stage LV_K may be smaller by one (e.g., Q=P-1) than the number of the plurality of multipliers 151_1 to 151_P provided at the first stage LV_1.

[0058] A plurality of adders 152_1 to 154 may perform addition calculations on two data input thereto. For example, the first top adder 152_1 may perform addition calculations on data input from the first multiplier 151_1 and data input from the second multiplier 151_2. The first top adder 152_1 may output the result of the addition calculation to the third-level adder. In an embodiment, a plurality of multipliers 151_1 to 151_P and a plurality of adders 152_1 to 154 of the first calculation channel 211_1 may perform multiplication calculations and / or addition calculations, and the final adder 154 may output a calculation result OPR, that is, the final result of the multiplication calculation or addition calculation. The second calculation channel 211_2 to the mth calculation channel 211_m may also output calculation results. The calculation circuit 211 may send the calculation results of the first calculation channel 211_1 to the mth calculation channel 211_m to the plurality of storage devices 220.

[0059] Figure 5 is a block diagram of a CXL memory according to an embodiment.

[0060] refer to Figure 5 According to an embodiment, the CXL memory 500 may include a CXL controller 510 and a plurality of storage devices (e.g., a first storage device 571, a second storage device 572, a third storage device 573 to an nth storage device 574, where n is a positive integer greater than zero (0)). The plurality of storage devices 571 to 574 may store LLM data. The CXL controller 510 may be configured to receive LLM data from a host (e.g., Figure 1 The CXL controller 510 may load data from the plurality of storage devices 571 to 574 and perform calculations according to instructions of the host 110. For example, the CXL controller 510 may be connected to the host through the CXL interface 300 to receive instructions from the host.

[0061] CXL memory 500 may include and / or may be similar in many respects to the above-referenced Figure 1 and Figure 2 The CXL memories 130 and 200 described above may include additional features not mentioned above. In addition, the CXL controller 510 and the plurality of storage devices 571 to 574 may each include and / or may be similar in many respects to the CXL memories 130 and 200 described above. Figure 1 and Figure 2 The CXL controllers 131 and 210 and the plurality of storage devices 132 and 220 described above may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference may be omitted. Figure 1 and Figure 2 The description of the CXL memory 500 , the CXL controller 510 , and the plurality of storage devices 571 to 574 is repeated.

[0062] The CXL controller 510 may include a command buffer (BUF) 520 , a control logic (CONT) 530 , a computation circuit (CORE) 540 , a memory 550 , and a memory controller 560 .

[0063] Command buffer 520 may receive and store instructions from the host through CXL interface 300. For example, command buffer 520 may receive instructions from the host through the CXL.io protocol of CXL interface 300. As another example, the instructions of the host may include a command instructing to perform a matrix-vector multiplication calculation.

[0064] The control logic 530 may generate an internal signal based on the command stored in the command buffer 520. For example, the control logic 530 may generate an internal signal instructing the calculation circuit 540 to perform a matrix-vector multiplication calculation.

[0065] Memory 550 may be a memory inside CXL controller 510 and may be implemented as an on-chip memory. Memory 550 may receive instructions from the host via the CXL.mem protocol of CXL interface 300. For example, the host's instruction may include a logical address of input data and / or a logical address of LLM data.

[0066] The memory 550 may include a storage unit storing data, and an instruction circuit (logic) sending instructions to the storage controller 560. For example, the storage unit may store data received from a plurality of storage devices 571 to 574 through the storage controller 560. In an embodiment, the storage unit may be implemented with a static random access memory (SRAM). However, the present disclosure is not limited thereto, and other types of storage devices may be used to implement the storage unit.

[0067] The instruction circuit may send a read command to the memory controller 560 based on the instruction received from the host through the CXL.mem protocol. In some embodiments, the instruction circuit may be provided in various ways, for example, separately provided outside the memory 550 and / or integrated with the memory controller 560.

[0068] The storage controller 560 may control a plurality of storage devices 571 to 574. According to an embodiment, the storage controller 560 may be implemented as a plurality of storage controllers so that the plurality of storage controllers may control the plurality of storage devices 571 to 574. In an embodiment, the number of the storage controllers 560 may be the same as the number of the plurality of storage devices 571 to 574, and the storage controller 560 and the plurality of storage devices 571 to 574 may have a one-to-one correspondence.

[0069] The memory controller 560 may load input data and / or LLM data from the plurality of memory devices 571 to 574 based on a read command of the memory 550. The memory 550 may output a read command to the memory controller 560 based on an instruction of the host.

[0070] The memory controller 560 may store the LLM data in the memory 550. The memory 550 may store the LLM data input from the plurality of memory devices 571 to 574.

[0071] The calculation circuit 540 may perform calculations on the LLM data stored in the memory 550. For example, the calculation circuit 540 may include a Figure 4 The first to m-th calculation channels 211_1 to 211_m are described, and matrix-vector multiplication calculations are performed on LLM data stored in the memory 550 by using the first to m-th calculation channels 211_1 to 211_m. That is, the CXL memory 500 according to the embodiment can achieve high memory bandwidth and large memory capacity by using a plurality of memory devices 571 to 574, and efficiently perform matrix-vector multiplication calculations by using a calculation circuit 540 that may include a plurality of adder trees, thereby potentially reducing latency and power consumption when compared with related memory devices.

[0072] Figure 6is a schematic block diagram of a CXL memory according to an embodiment. Figure 7 is a diagram of a matrix-matrix multiplication computation performed by computation circuitry of a CXL memory, according to an embodiment. Figure 8 is a diagram of calculations of a calculation circuit according to an embodiment. Fig. 9 is a circuit diagram of a processing element included in a computing circuit according to an embodiment.

[0073] refer to Figure 6 , a CXL memory 600 according to an embodiment may include a CXL controller 610 and a plurality of storage devices 620. The plurality of storage devices 620 may include first to nth storage devices 620_1 to 620_n storing LLM data. The CXL controller 610 may include a first calculation circuit (CORE1) 611 and a second calculation circuit (CORE2) 612 that perform calculations on LLM data retrieved from the plurality of storage devices 620.

[0074] CXL memory 600 may include and / or may be similar in many respects to the above-referenced Figure 1 , Figure 2 and Figure 5 The CXL memories 130, 200, and 500 described above may include additional features not mentioned above. In addition, the CXL controller 610 and the plurality of storage devices 620 may each include and / or may be similar in many respects to those described above with reference to FIG. Figure 1 , Figure 2 and Figure 5 The CXL controllers 131, 210, and 510 and the plurality of storage devices 132, 220, and 571 to 574 described above may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference may be omitted. Figure 1 , Figure 2 and Figure 5 The description of CXL memory 600 , CXL controller 610 , and plurality of storage devices 620 is repeated.

[0075] According to an embodiment, the CXL controller 610 may receive an instruction from the host. For example, the CXL controller 610 may receive a command from the host through the CXL.io protocol 421. The CXL controller 610 may select the first computing circuit 611 and / or the second computing circuit 612 based on the command of the host. For example, if the host instructs a matrix-matrix multiplication calculation, the CXL controller 610 may select the first computing circuit 611. As another example, if the host instructs a matrix-vector multiplication calculation, the CXL controller 610 may select the second computing circuit 612.

[0076] The first computing circuit 611 may perform a matrix-matrix multiplication calculation on input data represented as a matrix and LLM data represented as a matrix. In an embodiment, the input data and the LLM data may be stored in a plurality of storage devices 620 and moved to the first computing circuit 611 for calculation. In an embodiment, the input data may be input to the first computing circuit 611 from the host through the CXL.mem protocol 422, and the LLM data may be stored in a plurality of storage devices 620 and moved to the first computing circuit 611 for calculation. The first computing circuit 611 may perform as described in reference Figure 7 Describes the matrix-matrix multiplication calculation. Refer to Figure 6 and Figure 7 , the first calculation circuit 611 may perform a multiplication calculation on the input data INPUT and the LLM data WEIGHT to generate the output data OUTPUT. The first calculation circuit 611 may include a plurality of processing elements that perform a multiplication calculation on a matrix of size HT×WD (e.g., HT rows and WD columns) among the input data INPUT and a matrix of size WD×HT (e.g., WD rows and HT columns) among the LLM data WEIGHT, where HT and WD are positive integers greater than zero (0). The number of the plurality of processing elements included in the first calculation circuit 611 may be HT×WD. For example, in the first calculation circuit 611, WD processing elements may be arranged in a horizontal direction, HT processing elements may be arranged in a vertical direction, LLM data WEIGHT may be input in a horizontal direction, and the input data INPUT may be input in a vertical direction. According to embodiments, the arrangement method of the plurality of processing elements and the input method of the input data INPUT and the LLM data WEIGHT may be implemented differently.

[0077] return Figure 6 , the second computation circuit 612 may perform a matrix-vector multiplication computation on input data represented as a vector and LLM data represented as a matrix. In an embodiment, the input data and the LLM data may be stored in a plurality of storage devices 620 and moved to the second computation circuit 612 for computation. In an embodiment, the input data may be input from the host to the second computation circuit 612 via the CXL.mem protocol 422, and the LLM data may be input to the plurality of storage devices 620 and moved to the second computation circuit 612 for computation. The second computation circuit 612 may perform as described in reference Figure 3 The second computation circuit 612 may include and / or may be similar in many respects to Figure 4 Therefore, for the sake of brevity, the description of the second calculation circuit 612 is omitted.

[0078] refer to Figure 8, the first calculation circuit 611 may receive input data INPUT and LLM data WEIGHT. The input data INPUT and LLM data WEIGHT may be input to the first calculation circuit 611 through the buffer memory. The first calculation circuit 611 may perform a matrix-matrix multiplication calculation on the input data INPUT and the LLM data WEIGHT. The input data INPUT may include first to fourth input data DIN1 to DIN4, and the LLM data WEIGHT may include first to fourth LLM data WT1 to WT4. The first calculation circuit 611 may include a plurality of processing elements (e.g., processing elements 161_1 to 164_4). Among the plurality of processing elements 161_1 to 164_4, WD processing elements may be arranged in a horizontal direction (e.g., an x ​​direction), HT processing elements may be arranged in a vertical direction (e.g., a y direction), and the first calculation circuit 611 may include a total of WD×HT processing elements 161_1 to 164_4.

[0079] like Figure 8 As shown, the plurality of processing elements 161_1 to 164_4 may be arranged in a systolic array structure. However, the present disclosure is not limited thereto. That is, the plurality of processing elements 161_1 to 164_4 may be arranged in different topological structures, such as but not limited to a linear array structure, a ring structure, a star structure, a tree structure, a mesh structure, a fully connected structure, a chord ring structure, a three-dimensional cube structure, etc.

[0080] although Figure 8 The first computing circuit 611 is depicted as an output smooth (OS) data flow method, but the present disclosure is not limited to this, and the first computing circuit 611 can be implemented using other data flow methods, such as but not limited to a weight smooth (WS) method, an input smooth (IS) method, a row smooth (RS) method, a no local reuse (NLR) method, etc.

[0081] The first calculation circuit 611 may operate based on a clock signal. For example, the first calculation circuit 611 may sequentially input input data INPUT and LLM data WEIGHT for each cycle of the clock signal. The cycle may represent an oscillation cycle of the clock signal. For example, the first calculation circuit 611 may input first data (e.g., "a") of the first input data DIN1 and first data (e.g., "d") of the first LLM data WT1 to the processing element 161_1 in the first cycle of the clock signal. The first calculation circuit 611 may input second data (e.g., "b") of the first input data DIN1 and second data (e.g., "e") of the first LLM data WT1 to the processing element 161_1 in the second cycle after the first cycle of the clock signal. That is, the processing element 161_1 may perform calculations on the first data (e.g., "a") of the first input data and the first data (e.g., "d") of the first LLM data in the first cycle, and perform calculations on the second data (e.g., "b") of the first input data and the second data (e.g., "e") of the first LLM data in the second cycle.

[0082] The processing element 161_1 may send the first data (e.g., “a”) of the first input data to the processing element 161_2 in the second cycle, and send the first data (e.g., “d”) of the first LLM data to the processing element 162_1. The first calculation circuit 611 may input the first data (e.g., “c”) of the second input data DIN2 to the processing element 162_1 in the second cycle. The processing element 162_1 may perform calculations on the first data (e.g., “c”) of the second input data and the first data (e.g., “d”) of the first LLM data in the second cycle. The first calculation circuit 611 may input the first data (e.g., “f”) of the second LLM data WT2 to the processing element 161_2 in the second cycle. The processing element 161_2 may perform calculations on the first data (e.g., “a”) of the first input data and the first data (e.g., “f”) of the second LLM data in the second cycle.

[0083] In an embodiment, the plurality of processing elements 161_1 to 164_4 of the first computing circuit 611 may perform matrix-matrix multiplication calculations on the input data INPUT and the LLM data WEIGHT and output output data OUTPUT of the calculation results. The first computing circuit 611 may send the output data OUTPUT to the plurality of storage devices 620 through the buffer memory.

[0084] Reference together Figure 8 and Fig. 9According to an embodiment, the processing element 161_1 may include a multiplier 171, an adder 172, and an accumulator (ACC) 173. The processing element 161_1 may receive first data (e.g., "a") of first input data and first data (e.g., "d") of first LLM data in a first cycle of a clock signal. The multiplier 171 may generate a first calculation value by performing a multiplication calculation on the first data (e.g., "a") of the first input data and the first data (e.g., "d") of the first LLM data. The multiplier 171 may output the first calculation value to the adder 172. The adder 172 may perform an addition calculation on the calculation value output by the multiplier 171 and the accumulated value output by the accumulator 173. In the first cycle, there is no accumulated value output by the accumulator 173, so the adder 172 may send the first calculation value to the accumulator 173. The accumulator 173 may store the first calculation value. Processing element 1611 may send first data (e.g., “a”) of the first input data to processing element 1612 in the second cycle, and send first data (e.g., “d”) of the first LLM data to processing element 1621. Processing element 1612 and processing element 1621 may perform calculations in a manner similar to processing element 1611.

[0085] The processing element 1611 may receive the second data (e.g., "b") of the first input data and the second data (e.g., "e") of the first LLM data input in the second cycle of the clock signal. The multiplier 171 may generate a second calculation value by performing a multiplication calculation on the second data (e.g., "b") of the first input data and the second data (e.g., "e") of the first LLM data. The multiplier 171 may output the second calculation value to the adder 172. The adder 172 may generate a th accumulated value by performing an addition calculation on the second calculation value output by the multiplier 171 and the first calculation value output by the accumulator 173. The adder 172 may send the th accumulated value to the accumulator 173. The accumulator 173 may store the first accumulated value. The accumulator 173 may send the first accumulated value to the adder 172 in a third cycle after the second cycle. When the calculation of the processing element 1611 is completed, the adder 172 may output the stored accumulated value PSM. The accumulated values ​​of the plurality of processing elements 1611 to 1644 may be sent as output data OUTPUT to a plurality of storage devices.

[0086] Fig.10 is a block diagram of a CXL memory according to an embodiment.

[0087] refer to Fig.10According to an embodiment, the CXL memory 800 may include a CXL controller 810 and a plurality of storage devices (e.g., a first storage device 871, a second storage device 872, a third storage device 873 to an nth storage device 874, where n is a positive integer greater than zero (0)). The plurality of storage devices 871 to 874 may store LLM data. According to an embodiment, input data may be stored in the plurality of storage devices 871 to 874 and / or may be received from a host to the CXL controller 810. The CXL controller 810 may perform calculations by loading data from the plurality of storage devices 871 to 874 according to an instruction of the host. For example, the CXL controller 810 may be connected to the host through the CXL interface 700 to receive instructions from the host.

[0088] CXL controller 810 may include command buffer (BUF) 820, control logic (CONT) 830, compute engine 840, memory 850, and storage controller 860. Command buffer 820, control logic 830, memory 850, and storage controller 860 may include and / or may be similar in many respects to Figure 5 The command buffer 520, control logic 530, memory 550 and storage controller 560 of FIG. 50 may include additional features not mentioned above. Therefore, for the sake of brevity, the above reference may be omitted. Figure 5 The description of command buffer 820, control logic 830, memory 850, and memory controller 860 is repeated.

[0089] The host may send a command indicating a computation to CXL controller 810 via CXL interface 700. The computation may include a matrix-matrix multiplication computation and / or a matrix-vector multiplication computation. Command buffer 820 may store host commands. Control logic 830 may generate internal signals based on the commands stored in command buffer 820. For example, if the command stored in command buffer 820 is related to a matrix-matrix multiplication computation, control logic 830 may generate a first signal. Alternatively or additionally, if the command stored in command buffer 820 is related to a matrix-vector multiplication computation, control logic 830 may generate a second signal.

[0090] The computing engine 840 may include a first computing circuit (CORE1) 841 and a second computing circuit (CORE2) 842. The first computing circuit 841 and the second computing circuit 842 may include and / or may be similar in many respects to Figure 6The computing engine 840 may select the first computing circuit 841 and / or the second computing circuit 842 based on the internal signal generated by the control logic 830. For example, if the control logic 830 generates a first signal, the computing engine 840 selects the first computing circuit 841. Alternatively or additionally, if the control logic 830 generates a second signal, the computing engine 840 may select the second computing circuit 842. That is, the first computing circuit 841 may perform a matrix-matrix multiplication calculation, and the second computing circuit 842 may perform a matrix-vector multiplication calculation.

[0091] The selected computation circuit (e.g., the first computation circuit 841 and / or the second computation circuit 842) may load at least one of input data and LLM data from a plurality of storage devices 871 to 874. For example, the memory controller 860 may load at least one of input data and LLM data from a plurality of storage devices 871 to 874 to store in the memory 850. The selected computation circuit may receive input data from a host to store it in the memory 850. The selected computation circuit may perform a matrix-matrix multiplication computation and / or a matrix-vector multiplication computation on the input data and LLM data stored in the memory 850.

[0092] That is, according to an embodiment, the CXL memory 800 can achieve high memory bandwidth and large memory capacity by using a plurality of memory devices 871 to 874, and can select a first computing circuit 841 that can perform matrix-matrix multiplication calculations and / or a second computing circuit 842 that can perform matrix-vector multiplication calculations to perform efficient calculations, thereby potentially reducing latency and energy consumption when compared to related memory devices.

[0093] Fig.11 is a block diagram of a processing engine according to an embodiment.

[0094] refer to Fig.11 According to an embodiment, the processing engine 1100 may include an interface 1110, a scheduler 1120, and engines (eg, a first engine 1130, a second engine 1140, a third engine 1150, and a fourth engine 1160).

[0095] The interface 1110 may receive data as a calculation object. For example, the data to be calculated may be input data and / or LLM data. The interface 1110 may receive data from a memory (e.g., an on-chip memory and / or an external storage device). The interface 1110 may send data to at least one of the first engine 1130 to the fourth engine 1160.

[0096] The scheduler 1120 may be based on the control logic (e.g., Fig.10 The scheduler 1120 may select the first engine 1130 or the second engine 1140 based on an internal signal generated by the control logic 830 in the control logic 830. For example, the scheduler 1120 may select the first engine 1130 when the control logic outputs the first signal, and select the second engine 1140 when the control logic outputs the second signal. The first signal may correspond to a matrix-matrix multiplication calculation instruction of the host, and the second signal may correspond to a matrix-vector multiplication calculation instruction of the host. That is, the first engine 1130 may perform a matrix-matrix multiplication calculation, and the second engine 1140 may perform a matrix-vector multiplication calculation.

[0097] The first engine 1130 may include a first computing circuit (CORE1) 1131 and a first register (REG1) 1132. The first computing circuit 1131 may include a processing element array. The processing element array may include WD processing elements in the horizontal direction and HT processing elements in the vertical direction. That is, the processing element array may include a total of WD×HT processing elements and perform calculations of input data of a matrix size of WD×HT. A processing element may include a multiplier and an adder. The first register 1132 may store input data for calculations of the first computing circuit 1131. The first register 1132 may receive input data from a memory and / or a host. The first computing circuit 1131 may perform a matrix-matrix multiplication calculation on the input data of the first register 1132 and the LLM data received from the storage device.

[0098] In an embodiment, the third engine 1150 and / or the fourth engine 1160 may be configured to perform as an auxiliary engine. For example, the third engine 1150 may perform preprocessing on at least one of the input data and the LLM data for calculation by the first engine 1130. In an embodiment, the third engine 1150 may include at least one of a transpose circuit for calculating the transpose of a matrix and an image to column (im2col) circuit for converting multidimensional data into matrix data. The third engine 1150 may send the preprocessed data to the first engine 1130.

[0099] The fourth engine 1160 may include a circuit for performing calculations other than matrix-matrix multiplication calculations and matrix-vector multiplication calculations. For example, the fourth engine 1160 may perform calculations of nonlinear functions and may include at least one of a residual circuit, a Gaussian error linear unit (GELU) circuit, and a normalization (standardization) circuit. The residual circuit may combine the previous calculation result with the current calculation result. For example, the residual circuit may combine the previous calculation result and the current calculation result present at the same position. The previous calculation result and the current calculation result may include data present at the same position in space. The GELU circuit may perform calculations on an exponential function. The normalization circuit may perform equalization on the data. The fourth engine 1160 may send the calculation result to the first engine 1130 and / or the second engine 1140.

[0100] The second engine 1140 may include a second computing circuit (CORE2) 1141 and a second register (REG2) 1142. The second computing circuit 1141 may include a plurality of adder trees. For example, the second computing circuit 1141 may include u channels, where u is a positive integer greater than zero (0). Alternatively or additionally, one channel may include one adder tree. The adder tree may include a plurality of multipliers at the first stage of input data, and the plurality of adders may be set from the second stage, which is the next stage after the first stage, to the final stage of outputting the calculation result. In the adder tree of one channel, the number of multipliers may be v (where v is a positive integer greater than zero (0)), and the number of adders may be less than the number of multipliers (for example, v-1). That is, the number of multipliers in the second computing circuit 1141 may be u×v, and the number of adders may be u×(v-1). The second computing circuit 1141 may perform matrix-vector multiplication calculations by using a plurality of adder trees. The second register 1142 may store input data for calculation of the second calculation circuit 1141. The second register 1142 may receive input data from a memory or a host. The second calculation circuit 1141 may perform a matrix-vector multiplication calculation on the input data of the second register 1142 and the LLM data received from the storage device.

[0101] Fig.12 is a block diagram of a computing system using CXL memory according to an embodiment.

[0102] refer to Fig.12 According to an embodiment, the computing system 1200 may be and / or may include a server or a data center. The computing system 1200 may provide services to multiple tenants. In an embodiment, the computing system 1200 may provide artificial intelligence (AI) services.

[0103] According to an embodiment, the computing system 1200 may include a host 1210, a first CXL memory 1230, and a second CXL memory 1240. The first CXL memory 1230 may provide a service to a first tenant among a plurality of tenants by storing first LLM data, and the second CXL memory 1240 may provide a service to a second tenant among a plurality of tenants by storing second LLM data. The host 1210, the first CXL memory 1230, and the second CXL memory 1240 may be connected and communicated through a CXL interface 1220. The CXL interface 1220 may include a lower layer protocol, such as, but not limited to, a CXL.io protocol, a CXL.mem protocol, and a CXL.cache protocol. The first CXL memory 1230 and the second CXL memory 1240 may communicate with each other through the CXL interface 1220.

[0104] The host 1210 may control the overall operation of the computing system 1200. In an embodiment, the host 1210 may be and / or may include one of various processors, such as but not limited to a CPU, a GPU, an NPU, a DPU, etc. In an embodiment, the host 1210 may include a single-core processor or a multi-core processor. The host 1210 may output a command indicating a calculation to the first CXL memory 1230 and / or the second CXL memory 1240. For example, the host 1210 may output a first command to the first CXL memory 1230 and a second command to the second CXL memory 1240. The host 1210 may instruct a matrix-matrix multiplication calculation and / or a matrix-vector multiplication calculation. The host 1210 may send a command by using the CXL.io protocol of the CXL interface 1220. The first CXL memory 1230 may perform a calculation based on the first command and provide an AI service to the first tenant. The second CXL memory 1240 may perform a calculation based on the second command and provide an AI service to the second tenant.

[0105] The first CXL memory 1230 may include a CXL controller 1231 and a plurality of storage devices 1232. The CXL controller 1231 may store data in the plurality of storage devices 1232 and / or transmit data stored in the plurality of storage devices 1232 to the host 1210 according to the control of the host 1210.

[0106] The plurality of storage devices 1232 may include first storage devices 1232_1 to ath storage devices 1232_a, and may store first LLM data, where a is a positive integer greater than zero (0). The CXL controller 1231 may load the first LLM data of the plurality of storage devices 1232 according to a first command of the host 1210, and perform calculation. In an embodiment, the plurality of storage devices 1232 may be and / or may include DRAM. However, the present disclosure is not limited thereto. That is, the plurality of storage devices 1232 may be and / or may include other types of storage devices.

[0107] The second CXL memory 1240 may include a CXL controller 1241 and a plurality of storage devices 1242. The plurality of storage devices 1242 may include first to b-th storage devices 1242_1 to 1242_b, and may store second LLM data, where b is a positive integer greater than zero (0). The CXL controller 1241 may perform calculations on the second LLM data based on a second command of the host 1210. Figures 1 to 11 The described embodiments of the CXL memory may be similarly applied to the first CXL memory 1230 and the second CXL memory 1240. For example, the first CXL memory 1230 and the second CXL memory 1240 may include one or more computation circuits. The computation circuit may include a processing element array for performing matrix-matrix multiplication computations and / or an adder tree for performing matrix-vector multiplication computations. If the first CXL memory 1230 and the second CXL memory 1240 include two computation circuits, the first CXL memory 1230 and the second CXL memory 1240 may select a computation circuit based on an instruction from the host 1210. In addition, it should be understood that the CXL memory of different embodiments may be similarly applied to the first CXL memory 1230 and the second CXL memory 1240.

[0108] In an embodiment, the host 1210, the first CXL memory 1230, and the second CXL memory 1240 may be configured to share the same interface with each other. For example, the host 1210, the first CXL memory 1230, and the second CXL memory 1240 may communicate with each other through the CXL interface 1220. In an embodiment, the CXL interface 1220 may refer to a low-latency and high-bandwidth link that can implement various connections between accelerators, storage devices, or various electronic devices by supporting memory coherency, multiplexed memory access, and dynamic protocol multiplexing of input / output protocols.

[0109] According to an embodiment, the host 1210 may provide LLM services to multiple tenants by using the first CXL memory 1230 and the second CXL memory 1240, wherein the first CXL memory 1230 and the second CXL memory 1240 support high memory bandwidth and large memory capacity by using the CXL interface 1220. That is, according to an embodiment, the computing system 1200 may shorten the movement time of LLM data and reduce the overhead in terms of execution time and energy by applying the first CXL memory 1230 and the second CXL memory 1240 that respectively store LLM data. Alternatively or additionally, according to an embodiment, the computing system 1200 may select an appropriate computing circuit according to the computing type of the LLM, so that efficient computing may be performed with low latency and low energy.

[0110] Fig.13 is a block diagram of a computing system using CXL memory according to an embodiment.

[0111] refer to Fig.13 According to an embodiment, the computing system 1300 may be a user device and / or may be included in a user device, such as but not limited to a PC, a laptop computer, a server, a data center, a media player, a digital camera, etc., and / or may be a vehicle device and / or may be included in a vehicle device, such as but not limited to a navigation system, a black box, a vehicle electronic device, etc. Alternatively or additionally, the computing system 1300 may be and / or may include a mobile system, such as but not limited to a portable communication terminal, a smart phone, a tablet PC, a wearable device, a healthcare device, an IoT device, etc. The computing system 1300 may provide AI services.

[0112] According to an embodiment, the computing system 1300 may include a host 1310, an accelerator 1330, and a CXL memory 1340. The accelerator 1330 and the CXL memory 1340 may provide AI services by performing calculations on LLM data. The accelerator 1330 may perform calculations on first LLM data, and the CXL memory 1340 may perform calculations on second LLM data. According to an embodiment, the first LLM data and the second LLM data may be the same or may be different. That is, the accelerator 1330 and the CXL memory 1340 may perform calculations on the same LLM data separately, and / or perform calculations on different LLM data respectively. The host 1310, the accelerator 1330, and the CXL memory 1340 may be connected and communicated through a CXL interface 1320. The CXL interface 1320 may include lower layer protocols, such as, but not limited to, CXL.io protocol, CXL.mem protocol, and CXL.cache protocol. The accelerator 1330 and the CXL memory 1340 may communicate with each other through the CXL interface 1320.

[0113] The host 1310 may control the overall operation of the computing system 1300. In an embodiment, the host 1310 may be and / or may include one of various processors, such as but not limited to a CPU, a GPU, an NPU, a DPU, etc. In an embodiment, the host 1310 may include a single-core processor or a multi-core processor. The host 1310 may output a command indicating a calculation to the accelerator 1330 and / or the CXL memory 1340. The host 1310 may indicate a matrix-matrix multiplication calculation and / or a matrix-vector multiplication calculation. The host 1310 may output a first command to the accelerator 1330 and output a second command to the CXL memory 1340. In an embodiment, the host 1310 may indicate a matrix-matrix multiplication calculation to the accelerator 1330 and indicate a matrix-vector multiplication calculation to the CXL memory 1340. In an embodiment, the host 1310 may indicate a matrix-matrix multiplication calculation to the accelerator 1330 and indicate a matrix-matrix multiplication calculation to the CXL memory 1340. In an embodiment, the host 1310 may indicate a matrix-matrix multiplication calculation to the accelerator 1330 and indicate a matrix-matrix multiplication calculation and / or a matrix-vector multiplication calculation to the CXL memory 1340. Alternatively or additionally, the accelerator 1330 may output a third command to the CXL memory 1340. For example, the accelerator 1330 may indicate a matrix-matrix multiplication calculation and / or a matrix-vector multiplication calculation to the CXL memory 1340. In an embodiment, when the accelerator 1330 receives a matrix-vector multiplication calculation from the host 1310, the accelerator 1330 may indicate the matrix-vector multiplication calculation to the CXL memory 1340. In an embodiment, if the accelerator 1330 receives a matrix-matrix multiplication calculation from the host 1310 and the available memory of the accelerator 1330 is below a threshold, the accelerator 1330 may indicate the matrix-matrix multiplication calculation to the CXL memory 1340. The accelerator 1330 may perform the calculation based on the first command, and the CXL memory 1340 may perform the calculation based on at least one of the second command and the third command. In an embodiment, the CXL memory 1340 may process the second command in priority to the third command.

[0114] The host 1310 may send a command to the accelerator 1330 and / or the CXL memory 1340 by using the CXL.io protocol of the CXL interface 1320. The accelerator 1330 may send a command to the CXL memory 1340 by using the CXL.io protocol of the CXL interface 1320. The accelerator 1330 may improve the performance of the computing system 1300 by distributing the workload of the host 1310 when compared to a related storage device. The accelerator 1330 may be and / or may include one of various processors, such as but not limited to a GPU, an NPU, a DPU, etc. The accelerator 1330 may perform a calculation on the first LLM data based on the first command of the host 1310. The accelerator 1330 may store the first LLM data in its memory, and the processor of the accelerator 1330 may load the first LLM data from the memory to perform the calculation.

[0115] The CXL memory 1340 may include a CXL controller 1341 and a plurality of storage devices 1342. The CXL controller 1341 may store data in the plurality of storage devices 1342 and / or transmit data stored in the plurality of storage devices 1342 to the host 1310 according to the control of the host 1310.

[0116] The plurality of storage devices 1342 may include a first storage device 1342_1 to a c-th storage device 1342_c, and may store second LLM data, where c is a positive integer greater than zero (0). The CXL controller 1341 may load the second LLM data of the plurality of storage devices 1342 according to a second command of the host 1310, and perform calculation. In an embodiment, the plurality of storage devices 1342 may be and / or may include DRAM. However, the present disclosure is not limited thereto. That is, the plurality of storage devices 1342 may be and / or may include other types of storage devices.

[0117] refer to Figures 1 to 11 The embodiments of the CXL memory described may be similarly applied to CXL memory 1340. For example, CXL memory 1340 may include one or more computation circuits. The computation circuit may include a processing element array for performing matrix-matrix multiplication computations and / or an adder tree for performing matrix-vector multiplication computations. If CXL memory 1340 includes two computation circuits, CXL memory 1340 may select a computation circuit based on an instruction from host 1310.

[0118] In an embodiment, the host 1310, the accelerator 1330, and the CXL memory 1340 may be configured to share the same interface with each other. For example, the host 1310, the accelerator 1330, and the CXL memory 1340 may communicate with each other via the CXL interface 1320. In an embodiment, the CXL interface 1320 may refer to a low-latency and high-bandwidth link that can implement various connections between accelerators, storage devices, or various electronic devices by supporting storage consistency, multiplexed memory access, and dynamic protocol multiplexing of input / output protocols.

[0119] According to an embodiment, the host 1310 or the accelerator 1330 may serve the LLM by using the CXL memory 1340, wherein the CXL memory 1340 supports high memory bandwidth and large memory capacity by using the CXL interface 1320. That is, according to an embodiment, the computing system 1300 may perform efficient computing by applying the accelerator 1330 and the CXL memory 1340, thereby potentially shortening the movement time of LLM data when compared with a related storage device, and reducing overhead in terms of execution time and energy. Alternatively or additionally, according to an embodiment, the CXL memory 1340 may select an appropriate computing circuit according to the computing type of the LLM, so that the computing system 1300 may perform efficient computing with low latency and low energy.

[0120] Fig.14 is a flowchart of a calculation method of a CXL memory according to an embodiment.

[0121] refer to Fig.14 According to an embodiment, the CXL memory may receive a command from a host (operation S1410). The host and the CXL memory may be connected via a CXL interface. The host may send a command to the CXL memory via a CXL.io protocol using the CXL interface. The host may instruct the CXL memory to perform at least one of a matrix-matrix multiplication calculation and a matrix-vector multiplication calculation.

[0122] The CXL memory may select a computing circuit based on a command of the host (operation S1420). The CXL memory may include a first computing circuit that performs a matrix-matrix multiplication calculation and a second computing circuit that performs a matrix-vector multiplication calculation. The first computing circuit may include a PE array in which a plurality of processing elements are arranged. One processing element may include a multiplier and an adder. In an embodiment, one processing element may include one multiplier and one adder, but the present disclosure is not limited thereto.

[0123] The second calculation circuit may include an adder tree in which a plurality of multipliers and a plurality of adders are configured to perform calculations in a tree shape. The plurality of multipliers may be provided at a first stage of input data of the adder tree, and the plurality of adders may be provided from a second stage to a final stage. At the final stage, the calculation result may be output. When the host indicates a matrix-matrix multiplication calculation, the CXL memory may select the first calculation circuit, and when the host indicates a matrix-vector multiplication calculation, the CXL memory may select the second calculation circuit.

[0124] The CXL memory may process the command by using the selected calculation circuit (operation S1430). For example, the CXL memory may perform a matrix-matrix multiplication calculation by using the first calculation circuit, and / or may perform a matrix-vector multiplication calculation by using the second calculation circuit. The CXL memory may load input data and LLM data from a storage device. According to an embodiment, the CXL memory may receive input data from a host and load LLM data from a storage device. In the CXL memory, the first calculation circuit and the second calculation circuit may be separately provided from the storage device. The selected calculation circuit may perform a multiplication calculation on the input data and the LLM data.

[0125] Fig.15 is a block diagram of a computing system to which a storage system according to an embodiment is applied.

[0126] refer to Fig.15 , the computing system 1500 may include a host 1510, a plurality of storage devices (e.g., a first storage device 1511 and a second storage device 1512), a CXL storage device 1520, and a CXL memory 1530. In an embodiment, the computing system 1500 according to the embodiment may be a user device and / or may be included in a user device, such as but not limited to a PC, a laptop computer, a server, a data center, a media player, a digital camera, etc., and / or may be a vehicle device and / or may be included in a vehicle device, such as but not limited to a navigation system, a black box, a vehicle electronic device, etc. Alternatively or additionally, the computing system 1500 may be and / or may include a mobile system, such as but not limited to a portable communication terminal, a smart phone, a tablet PC, a wearable device, a healthcare device, an IoT device, etc. The computing system 1500 may provide an AI service.

[0127] The host 1510 may control the overall operation of the computing system 1500. In an embodiment, the host 1510 may be and / or may include one of various processors, such as but not limited to a CPU, a GPU, an NPU, a DPU, etc. In an embodiment, the host 1510 may include a single-core processor or a multi-core processor.

[0128] The plurality of storage devices 1511 and 1512 may be used as main memory and / or system memory of the computing system 1500. In an embodiment, each of the plurality of storage devices 1511 and 1512 may be a DRAM device and may have a dual in-line memory module (DIMM) form factor. However, the scope of the present disclosure is not limited in this regard, and the plurality of storage devices 1511 and 1512 may include non-volatile memory, such as flash memory, parametric random access memory (PRAM), resistive random access memory (RRAM), magnetoresistive random access memory (MRAM), etc.

[0129] The plurality of storage devices 1511 and 1512 may communicate directly with the host 1510 through a double data rate (DDR) interface. In an embodiment, the host 1510 may include a storage controller configured to control the plurality of storage devices 1511 and 1512. However, the present disclosure is not limited thereto, and the plurality of storage devices 1511 and 1512 may communicate with the host 1510 through various other interfaces.

[0130] The CXL storage device 1520 may include a CXL storage controller 1521 and a non-volatile memory (NVM). The CXL storage controller 1521 may store data in the NVM and / or transmit data stored in the NVM to the host 1510 and / or the CXL memory 1530 according to the control of the host 1510. The NVM may store LLM data, and the NVM may maintain the LLM data even when the CXL storage device 1520 is turned off. In an embodiment, the NVM may be a NAND flash memory. However, the scope of the present disclosure is not limited thereto. That is, the NVM may be and / or may include other types of storage devices.

[0131] The CXL memory 1530 may include a CXL memory controller 1531 and a buffer memory BFM. The CXL memory controller 1531 may store data in the buffer memory BFM and / or transmit the data stored in the buffer memory BFM to the host 1510 according to the control of the host 1510. For example, the CXL memory controller 1531 may store data of the plurality of storage devices 1511 and 1512 or the CXL memory device 1520 in the buffer memory BFM according to the control of the host 1510. In an embodiment, the buffer memory BFM may be and / or may include a DRAM. However, the present disclosure is not limited thereto. That is, the buffer memory BFM may be and / or may include other types of storage devices.

[0132] refer to Figures 1 to 11 The embodiments of the CXL memory described may be similarly applied to CXL memory 1530. For example, CXL memory 1530 may include one or more computation circuits. The computation circuit may include a processing element array for performing matrix-matrix multiplication computations and / or an adder tree for performing matrix-vector multiplication computations. If CXL memory 1530 includes two computation circuits, CXL memory 1530 may select a computation circuit based on an instruction from host 1510.

[0133] In an embodiment, the host 1510, the CXL storage device 1520, and the CXL memory 1530 may be configured to share the same interface. For example, the host 1510, the CXL storage device 1520, and the CXL memory 1530 may communicate with each other via a CXL interface 1550. In an embodiment, the CXL interface 1550 may refer to a low-latency and high-bandwidth link that can implement various connections between accelerators, storage devices, or various electronic devices by supporting storage consistency, multiplexed memory access, and dynamic protocol multiplexing of input / output protocols.

[0134] In an embodiment, the CXL storage device 1520 may access the CXL memory 1530 through the CXL interface 1550. For example, the CXL storage device 1520 may store LLM data in an allocated area among areas of the CXL memory 1530, or read the stored LLM data. According to the control of the CXL storage device 1520, the CXL memory 1530 may store data (e.g., LLM data) in the buffer memory BFM and / or transmit the data (e.g., LLM data) stored in the buffer memory BFM to the CXL storage device 1520.

[0135] In an embodiment, a storage controller of the relevant storage device may communicate with the host 1510 via a host interface (e.g., Peripheral Component Interconnect Express (PCIe) and / or NVM Express (NVMe)), and / or may communicate with the buffer memory via a memory interface (e.g., but not limited to, a DDR or Low Power DDR (LPDDR) interface). That is, the storage controller of the relevant storage device may communicate with the externally placed host 1510 and the internally included buffer memory via different interfaces (e.g., heterogeneous interfaces).

[0136] In an embodiment, the CXL storage controller 1521 of the CXL storage device 1520 may communicate with the host 1510 and the CXL memory 1530 (e.g., a buffer memory) through the CXL interface 1550. That is, the CXL storage controller 1521 of the CXL storage device 1520 may communicate with the host 1510 and the CXL memory 1530 through a similar interface or a common interface, and may use some areas of the CXL memory 1530 as a buffer memory.

[0137] Although the host 1510, the CXL storage device 1520, and the CXL memory 1530 are described as communicating with each other through the CXL interface 1550, the present disclosure is not limited thereto. That is, the host 1510, the CXL storage device 1520, and the CXL memory 1530 may communicate with each other based on various other computing interfaces, such as, but not limited to, the Gen-Z protocol, the NVLink protocol, the CCIX protocol, and the open CAPI protocol.

[0138] Fig.16 Detailed description of the above components is omitted for better understanding and ease of description.

[0139] refer to Fig.16 , computing system 1600 may include a CXL switch 1605 , a first CPU 1610 , a second CPU 1620 , a GPU 1630 , an NPU 1640 , a CXL storage device 1650 , a CXL memory 1660 , a PCIe device 1670 , and an accelerator (eg, CXL device) 1680 .

[0140] The first CPU 1610 , the second CPU 1620 , the GPU 1630 , the NPU 1640 , the CXL storage device 1650 , the CXL memory 1660 , the PCIe device 1670 , and the accelerator 1680 may be commonly connected to the CXL switch 1605 , and may all communicate with each other through the CXL switch 1605 .

[0141] In an embodiment, each of the first CPU 1610, the second CPU 1620, the GPU 1630, and the NPU 1640 may be a reference Figures 1 to 15 1614, the fifth storage device 1615, the sixth storage device 1616, the seventh storage device 1617, and the eighth storage device 1618). In an embodiment, CXL storage device 1650 may be and / or may be similar in many respects to reference 1614. Fig.15 In an embodiment, CXL memory 1660 may be and / or may be similar in many respects to the CXL memory device described above, and may include additional features not mentioned above. Figures 1 to 11The CXL memory 1660 may include additional features not mentioned above. For example, CXL memory 1660 may include one or more computation circuits. The computation circuit may include an array of processing elements for performing matrix-matrix multiplication computations and / or an adder tree for performing matrix-vector multiplication computations. If CXL memory 1660 includes two computation circuits, CXL memory 1660 may select the computation circuit based on the host's instruction.

[0142] At least one region of CXL memory 1660 may be allocated as a dedicated region for CXL storage device 1650 by one or more of first CPU 1610, second CPU 1620, GPU 1630, and NPU 1640. That is, CXL storage device 1650 and CXL memory 1660 may be used as a storage space STR of computing system 1600.

[0143] In an embodiment, the CXL switch 1605 may be connected to a PCIe device 1670 and / or an accelerator 1680 configured to support various functions, and the PCIe device 1670 and / or the accelerator 1680 may communicate with the first CPU 1610 , the second CPU 1620 , the GPU 1630 , and the NPU 1640 , respectively, through the CXL switch 1605 , and / or access a storage space STR including a CXL storage device 1650 and a CXL memory 1660 .

[0144] In an embodiment, CXL switch 1605 may be connected to an external network (eg, network 1690 or Fabric), and may be configured to communicate with an external server through external network 1690 (or Fabric).

[0145] Fig.17 is a block diagram of a server to which a computing system according to an embodiment is applied.

[0146] refer to Fig.17, the server 2000 may be and / or may include a facility for collecting various data and providing services, and may be referred to as a data center or a data storage center. For example, the server 2000 may be a system for operating a search engine and a database, and may be a computing system used in a company and / or an entity (for example, but not limited to, a bank or a government agency). The server 2000 may include an application server (for example, a first application server 2110, a second application server 2120 to a d-th application server 2130, where d is a positive integer greater than zero (0)) and a storage server (for example, a first storage server 2210, a second storage server 2220 to an e-th storage server 2230, where e is a positive integer greater than zero (0)). According to an embodiment, the number of application servers 2110 to 2130 (for example, "d") and the number of storage servers 2210 to 2230 (for example, "e") may be selected in various ways. In an embodiment, the number of application servers 2110 to 2130 and the number of storage servers 2210 to 2230 may be different.

[0147] Hereinafter, the configuration of the first storage server 2210 will be mainly described. The application servers 2110 to 2130 and the storage servers 2210 to 2230 may all have similar structures, and the application servers 2110 to 2130 and the storage servers 2210 to 2230 may communicate with each other through the network (NT) 2205 .

[0148] The first storage server 2210 may include a processor 2211, a memory 2212, a switch 2213, a CXL memory 2214, a storage device 2215, and a network interface card (NIC) 2216. The processor 2211 may control the overall operation of the first storage server 2210, and may access the memory 2212 to execute commands loaded into the memory 2212 and / or process data. The memory 2212 may be and / or may include a DDR synchronous DRAM (DDR SDRAM), a high bandwidth memory (HBM), a hybrid memory cube (HMC), a DIMM, an Optane DIMM, and / or a non-volatile DIMM (NVM DIMM). The processor 2211 and the memory 2212 may be directly connected, and the number of processors 2211 and memories 2212 included in one storage server 2210 may be selected in various ways.

[0149] In an embodiment, the processor 2211 and the memory 2212 may provide a processor-memory pair. In an embodiment, the number of processors 2211 and the memory 2212 may be different. The processor 2211 may include a single-core processor or a multi-core processor. The above description of the storage server 2210 may be similarly applied to each of the application servers 2110 to 2130.

[0150] The switch 2213 may be configured to mediate and / or route communications between various configuration elements included in the first storage server 2210. In an embodiment, the switch 2213 may be a CXL switch implemented based on the CXL protocol.

[0151] CXL memory 2214 may be connected to switch 2213. In an embodiment, CXL memory 2214 may be used as a memory expander for processor 2211. Alternatively or additionally, CXL memory 2214 may be allocated as dedicated memory and / or buffer memory for storage device 2215.

[0152] refer to Figures 1 to 11 The embodiments of the CXL memory described may be similarly applied to CXL memory 2214. For example, CXL memory 2214 may include one or more computation circuits. The computation circuit may include a processing element array for performing matrix-matrix multiplication computations and / or an adder tree for performing matrix-vector multiplication computations. If CXL memory 2214 includes two computation circuits, CXL memory 2214 may select a computation circuit based on an instruction from processor 2211.

[0153] The storage device 2215 may include a CXL interface circuit (CXL_IF) 2217, a controller (CTRL) 2218, and a NAND flash memory (NAND) 2219. The storage device 2215 may store data and / or may output the stored data according to the request of the processor 2211. In an embodiment, the storage device 2215 may include and / or may be similar in many aspects to the reference Fig.15 and Fig.16 The CXL memory device described above may include additional features not mentioned above. In an embodiment, the storage device 2215 may allocate at least some areas of the CXL memory 2214 as a dedicated area, and use the dedicated area as a buffer memory.

[0154] A network interface card (NIC) 2216 may be connected to the switch 2213. The NIC 2216 may communicate with other storage servers 2220 to 2230 or other application servers 2110 to 2130 through the network 2205.

[0155] In an embodiment, the NIC 2216 may include a network interface card, a network adapter, etc. The NIC 2216 may be connected via a wired interface, a wireless interface, a BLUETOOTH interface, or a wireless interface. TMThe NIC 2216 may be connected to the network 2205 by a wireless interface, an optical interface, etc. The NIC 2216 may include an internal memory, a digital signal processor (DSP), a host bus interface, etc., and may be connected to the processor 2211 and / or the switch 2213, etc. through the host bus interface. In an embodiment, the NIC 2216 may be integrated with at least one of the processor 2211, the switch 2213, and the storage device 2215.

[0156] In an embodiment, the network 2205 can be implemented using a Fiber Channel (FC) and / or Ethernet network. As used herein, FC can refer to a medium for relatively high-speed data transmission, and an optical switch providing high performance / high availability can be used. Depending on the access method of the network 2205, storage servers 2210 to 2230 can be provided as a file storage device, a block storage device, or an object storage device.

[0157] In an embodiment, the network 2205 may be a storage-only network (e.g., but not limited to, a storage area network (SAN)). For example, the SAN may be an FC-SAN that uses an FC network and may be implemented according to the FC protocol (FCP). As another example, the SAN may be an Internet Protocol SAN (IP-SAN) that uses a Transmission Control Protocol / Internet Protocol (TCP / IP) network and may be implemented according to an Internet Small Computer System Interface (e.g., SCSI or iSCSI over TCP / IP) protocol. In an embodiment, the network 2205 may be a general network (e.g., a TCP / IP network). For example, the network 2205 may be implemented according to protocols such as, but not limited to, FC over Ethernet (FCoE), Network Attached Storage (NAS), and NVMe over Fabrics (NVMe-oF).

[0158] In an embodiment, at least one of the application servers 2110 to 2130 may store data requested by a user and / or a client to be stored in one of the storage servers 2210 to 2230 through the network 2205. At least one of the application servers 2110 to 2130 may obtain data requested by a user and / or a client to be read from one of the storage servers 2210 to 2230 through the network 2205. For example, at least one of the application servers 2110 to 2130 may be implemented as a network server and / or a database management system (DBMS).

[0159] In an embodiment, at least one of the application servers 2110 to 2130 may access a memory, a CXL memory, or a storage device included in another application server through the network 2205, or may access a memory, a CXL memory, or a storage device included in the storage servers 2210 to 2230 through the network 2205. Accordingly, at least one of the application servers 2110 to 2130 may perform various operations on data stored in other application servers and / or storage servers 2210 to 2230. For example, at least one of the application servers 2110 to 2130 may execute a command to move and / or copy data between other application servers and / or storage servers 2210 to 2230. In an embodiment, data may be moved from a storage device of a storage server 2210 to 2230 to a memory or CXL memory of an application server 2110 to 2130 through a memory and / or CXL memory of a storage server 2210 to 2230, and / or directly moved to a memory or CXL memory of an application server 2110 to 2130. In an embodiment, data moving over network 2205 may be encrypted data for security and / or privacy purposes.

[0160] In an embodiment, a storage device included in at least one of the application servers 2110 to 2130 and the storage servers 2210 to 2230 may be allocated with a CXL memory included in at least one of the application servers 2110 to 2130 and the storage servers 2210 to 2230 as a dedicated area, and the storage device may use the allocated dedicated area as a buffer memory. For example, a storage device 2215 included in the storage server 2210 may be allocated with a CXL memory included in another storage server (e.g., 2230), and the CXL memory included in another storage server (e.g., 2230) may be accessed through the switch 2213 and the NIC 2216. In such an example, LLM data of the storage device 2215 for the first storage server 2210 may be stored in the CXL memory of the other storage server 2230. That is, according to the present disclosure, the storage device and the CXL memory of the server 2000 may be connected and implemented in various ways.

[0161] In some embodiments, reference Figures 1 to 17 Each constituent element described, or a combination of two or more constituent elements may be implemented as a digital circuit, a programmable or non-programmable logic device or array, an application specific integrated circuit (ASIC), or the like.

[0162] While the disclosure has been described in connection with what are presently considered to be practical embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but on the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A Compute Express Link (CXL) storage device, include: Storage devices that store data; as well as Controller, configured as: reading data from the storage device based on a first command received via a first protocol; selecting a computing engine based on a second command received via a second protocol different from the first protocol; as well as The computing engine is controlled to perform computing on the read data.

2. The CXL storage device according to claim 1, in, The first protocol is the CXL.mem protocol, and Wherein, the second protocol is the CXL.io protocol.

3. The CXL storage device according to claim 1, in, The computing engine includes a first computing circuit and a second computing circuit, and Wherein, the controller is further configured as: selecting the first computation circuit based on the second command indicating a matrix-matrix multiplication computation; and The second computation circuit is selected based on the second command indicating a matrix-vector multiplication computation.

4. The CXL storage device according to claim 3, in, The first computing circuit is a processing element PE array, and Wherein, the second calculation circuit includes an adder tree.

5. The CXL storage device according to claim 1, in, The controller comprises: A control circuit configured to select the computing engine; a storage controller configured to read data from the storage device; and The on-chip memory is configured to store the read data and send the read data to the selected computing engine.

6. The CXL storage device according to claim 5, in, The controller further comprises: a command buffer configured to store the second command, The control circuit is further configured to generate a signal for selecting the computing engine based on the second command.

7. A computing system, include: a host configured to output at least one of a first computing command and a second computing command; an accelerator configured to operate based on the first computing command; a storage device configured to operate based on the second computing command; as well as Compute Fast Link CXL interface, configured as: sending the first computing command to the accelerator; as well as The second computing command is sent to the storage device.

8. The computing system according to claim 7, in, The CXL interface is further configured to send at least one of the first compute command and the second compute command via a CXL.io protocol.

9. The computing system according to claim 7, in, The first calculation command indicates a matrix-matrix multiplication calculation, and The second calculation command indicates a matrix-vector multiplication calculation.

10. The computing system according to claim 9, in, The storage device includes an adder tree, and The storage device is further configured to process the second computing command using the adder tree.

11. The computing system according to claim 10, in, The CXL interface is further configured to send the first computing command to the storage device, The storage device further comprises a processing element array configured to process the first computing command, and Wherein, the storage device is configured to use the processing element array to process the first computing command.

12. The computing system according to claim 11, in, The storage device further includes a control circuit, wherein the control circuit is configured to: selecting the array of processing elements based on receiving the first computing command; as well as The adder tree is selected based on receiving the second compute command.

13. The computing system according to claim 11, in, The accelerator is further configured to output a third computing command to the storage device based on the first computing command, and The storage device is further configured to use the processing element array to process the third computing command.

14. The computing system of claim 13, in, The CXL interface is further configured to send at least one of the first computing command, the second computing command, and the third computing command through a CXL.io protocol, and The storage device is further configured to process the first computing command in priority to the third computing command.

15. A Computational Express Link (CXL) storage device, include: Computational engine, including: A first calculation circuit configured to perform a first calculation on the input data and the large language model (LLM) data; and a second calculation circuit configured to perform a second calculation on the input data and the LLM data, the first calculation being different from the second calculation; and The scheduler is configured as: receiving computation commands via the CXL interface; and Based on the calculation command, at least one of the first calculation circuit and the second calculation circuit is selected.

16. The CXL storage device according to claim 15, in, The first computational circuit comprises an array of processing elements configured to perform matrix-matrix multiplication computations, wherein the second computation circuit comprises an adder tree configured to perform a matrix-vector multiplication computation, and Wherein, the scheduler is configured as follows: selecting the first calculation circuit based on the calculation command indicating the matrix-matrix multiplication calculation; and The second calculation circuit is selected based on the calculation command indicating the matrix-vector multiplication calculation.

17. The CXL storage device according to claim 15, further comprising: include: a storage device configured to store the input data and the LLM data, The calculation engine is configured to perform calculations on the input data and the LLM data by using at least one of the first calculation circuit and the second calculation circuit.

18. The CXL storage device according to claim 17, in, The scheduler is further configured to receive the computation command via the CXL.io protocol, and Wherein, the storage device is further configured as: Receive a logical address from the host via the CXL.mem protocol; and The input data and the LLM data are sent to the calculation engine based on the logical address.

19. The CXL storage device according to claim 18, in, The first computing circuit further includes a first register configured to store the input data, and The second calculation circuit further includes a second register configured to store the input data.

20. The CXL storage device of claim 15, further comprising: include: The third computing circuit is configured as follows: performing preprocessing on at least one of the input data and the LLM data; as well as sending the preprocessed data to the first computing circuit, The first calculation circuit is further configured to perform the first calculation based on the preprocessed data.