CXL device, electronic device and data storage method

Abstract

The invention provides a CXL device, an electronic device and a data storage method for managing memory resources. A CXL device according to an example embodiment may include: a memory configured to store first data; and a control logic configured to calculate a first usage ratio of the CXL device, and to output second data, which is part of first data stored in the CXL device, to the external shared memory through the CXL interface in response to the first usage ratio being greater than a reference value.

Description

Technical Field

[0001] The present disclosure relates to a CXL device, an electronic device, and a data storage method for efficiently managing memory resources. Background Art

[0002] The use of specialized workloads such as compression, encryption, and artificial intelligence, as well as the rapid growth of data, has increased the demand for heterogeneous computing, in which accelerators developed for special purposes work alongside general-purpose processors.

[0003] Accelerators require high-performance connectivity to processors, preferably sharing memory space to reduce overhead and latency. To this end, research has begun on chip-to-chip interconnect protocols that maintain memory and cache coherency by connecting processors to multiple accelerators. Summary of the invention

[0004] Example embodiments may provide a CXL device, an electronic device, and a data storage method for efficiently managing memory resources.

[0005] Example embodiments may provide a CXL device, an electronic device, and a data storage method for reducing data skew.

[0006] An embodiment provides a compute quick link (CXL) device, including: a memory configured to store first data; and a control logic configured to calculate a first usage ratio of the CXL device, and configured to output second data, which is a part of the first data stored in the CXL device, to an external shared memory through a CXL interface in response to the first usage ratio being greater than a reference value.

[0007] Another embodiment provides an electronic device including: a first CXL device configured to store second data as part of the first data in a second CXL device in response to a read request for the first data being greater than a reference number; a second CXL device configured to store the second data in response to a request of the first CXL device; and a switch configured to connect the first CXL device and the second CXL device.

[0008] Another embodiment provides a method for storing data, comprising: in response to a device receiving a first read request for first data from a host, sending the first data to the host; in response to the number of received first read requests being greater than a reference number, selecting one of the shared memories; determining second data as part of the first data based on the size of the first data, a first transmission rate of the selected shared memory, and a second transmission rate of the device; writing the second data to the selected shared memory; and in response to receiving a second read request for the first data from the host, requesting the selected shared memory to transmit the second data, and sending third data other than the second data among the first data to the host. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of a computing system is shown, according to some example embodiments.

[0010] Figures 2 to 6 A block diagram illustrating the operation of a computing system according to some example embodiments is shown.

[0011] Figure 7 An example of a mapping table managed by a first CXL storage device according to some example embodiments is shown.

[0012] Figure 8 An example of a mapping table managed by a second CXL storage device according to some example embodiments is shown.

[0013] Figures 9 to 12 A block diagram illustrating the operation of a computing system according to some example embodiments is shown.

[0014] Fig.13 A block diagram of a computing system is shown, according to some example embodiments.

[0015] Fig.14 A block diagram of a computing system is shown, according to some example embodiments.

[0016] Fig.15 A flow chart showing a method of storing data between CXL devices according to some example embodiments.

[0017] Fig.16 A flow chart showing a data transmission method of a CXL device according to some example embodiments is shown.

[0018] Fig.17 A flow chart of a data storage method according to some example embodiments is shown.

[0019] Fig.18 A block diagram of a computing system is shown, according to some example embodiments.

[0020] Fig.19A block diagram of a computing system is shown, according to some example embodiments.

[0021] Fig. 20 A block diagram of a server of an application computing system is shown according to some example embodiments. DETAILED DESCRIPTION

[0022] In the following detailed description, only certain example embodiments of the present disclosure are shown and described by way of illustration. As those skilled in the art will appreciate, the described example embodiments may be modified in many different ways, all without departing from the spirit or scope of the present disclosure.

[0023] Therefore, the drawings and description are to be considered illustrative rather than restrictive in nature, and the same reference numerals represent the same elements throughout the specification. In the flowcharts described with reference to the drawings in this specification, the order of operations may be changed, multiple operations may be combined, certain operations may be divided, and certain operations may not be performed.

[0024] In addition, singular forms may also include plural forms unless explicit expressions such as "one" or "single" are used. Terms including ordinal numbers such as first, second, etc. will only be used to describe a variety of constituent elements and should not be interpreted as limiting these constituent elements. These terms may be used to distinguish one constituent element from other constituent elements.

[0025] One or more elements disclosed below may include or be implemented in a processing circuit, such as hardware including a logic circuit; a hardware / software combination, such as a processor that executes software; or a combination thereof. For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc.

[0026] Figure 1 A block diagram of a computing system is shown, according to some example embodiments.

[0027] refer to Figure 1 , the computing system 10 may be included in a user device, such as a personal computer, laptop, server, data center, media player, or digital camera, or in an automotive device such as a GPS, black box, or vehicle electronics. Alternatively, the computer system 10 may be a mobile system, such as a mobile phone, smart phone, tablet personal computer, wearable device, healthcare device, or Internet of Things (IOT) device.

[0028] The computing system 10 may include hosts 11 to 14 and compute express link (CXL) devices 31 to 33. The hosts 11 to 14 and the CXL devices 31 to 33 may be connected to the cache coherence interface 20 through different physical ports. That is, when the CXL devices 31 to 33 are connected to the cache coherence interface 20, the memory area managed by the hosts 11 to 14 may have a high capacity, and the CXL devices 31 to 33 may exchange data with each other. The hosts 11 to 14 may include first to M-th hosts (M is an integer greater than 1). The CXL devices 31 to 33 may include first to N-th CXL devices (N is an integer greater than 1).

[0029] The cache coherent interface 20 may indicate a low latency and high bandwidth link that may support coherence of input / output (I / O) protocols, memory access, and dynamic protocol multiplexing to enable a variety of connections between accelerators, memory devices, or a variety of electronic devices.

[0030] In some example embodiments, the cache coherence interface 20 may be implemented with a CXL interface. The CXL interface may include a lower protocol, such as a CXL.io protocol, a CXL.mem protocol, or a CXL.cache protocol. The CXL.io protocol may include I / O semantics similar to PCIe. The CXL.cache protocol may include cache semantics, the CXL.mem protocol may include memory semantics, and the cache semantics and memory semantics may be optional. In some example embodiments, the hosts 11 to 14 may send instruction signals to the CXL devices 31 to 33 via the CXL.io protocol, and may receive data corresponding to the instruction signals via the CXL.io protocol. The CXL devices 31 to 33 may send and receive data to each other via the CXL.mem protocol.

[0031] The cache coherence interface 20 is not limited to the CXL interface, and the hosts 11 to 14 and the CXL devices 31 to 33 may communicate with each other based on various types of computing interfaces, such as the GEN-Z protocol, the NVLink protocol, the CCIX protocol, or the open coherent accelerator processor interface (CAPI) protocol.

[0032] Hosts 11 to 14 may control the general operation of computing system 10. In some example embodiments, hosts 11 to 14 may be one of a variety of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or a data processing unit (DPU). In some example embodiments, hosts 11 to 14 may include a single-core processor or a multi-core processor. The respective hosts 11 to 14 may be connected to at least one memory device via a double data rate (DDR) interface, and may send and receive data. Hosts 11 to 14 may include a memory controller for controlling at least one memory device. However, the scope of the present disclosure is not limited thereto, and hosts 11 to 14 may communicate with at least one memory device via multiple types of interfaces.

[0033] In some example embodiments, the hosts 11 to 14 may send a read request to the CXL devices 31 to 33, and the CXL devices 31 to 33 may send data to the hosts 11 to 14 based on the read request. In some example embodiments, the hosts 11 to 14 may divide a job, distribute the job to the CXL devices 31 to 33, and collect results. The hosts 11 to 14 may assign a job including data and a program (e.g., workload) for processing the data to the CXL devices 31 to 33.

[0034] Each CXL device 31 to 33 may be a memory device (or module) or a storage device (or module). The memory device may be a dynamic random access memory (DRAM) device and may have a variety of form factors, such as a dual in-line memory module (DIMM) or a high bandwidth memory (HBM). However, the scope of the present disclosure is not limited thereto, and the memory device may include a non-volatile memory, such as a flash memory, a phase change random access memory (PRAM), a resistive random access memory (RRAM), a magnetoresistive random access memory (MRAM), or a ferroelectric random access memory (FRAM). In addition, the storage device may be implemented with a variety of types of storage devices, such as a solid state drive (SSD), an embedded multimedia card (eMMC), a universal flash storage (UFS), a compact flash (CF), a secure digital (SD), a micro secure digital (Micro-SD), a mini secure digital (Mini-SD), an extreme digital (xD), or a memory stick.

[0035] When the hosts 11 to 14 simultaneously access a specific CXL device (e.g., the first CXL device 31) in the computing system 10, data skew may be generated to the first CXL device 31 due to a bottleneck phenomenon. The first CXL device 31 may move data to another CXL device (e.g., at least one of the second CXL device 32 to the Nth CXL device 33) to alleviate the data skew.

[0036] For example, the first CXL device 31 may move second data, which is a part of the frequently accessed large-volume first data, to the second CXL device 32. The second CXL device 32 may receive the second data from the first CXL device 31 and may store them. The first CXL device 31 may store third data other than the second data among the first data. For example, the third data may be data other than the second data among the first data. In response to a read request for the first data, the first CXL device 31 may output the third data, and the second CXL device 32 may output the second data. The read request may be an instruction sent by at least one of the hosts 11 to 14. Therefore, the computing system 10 may implement efficient space management through data dispersion, and may reduce the waiting time of the first data of the hosts 11 to 14.

[0037] The first CXL device 31 may calculate a usage ratio of the first CXL device 31. In some example embodiments, the usage ratio may indicate the operation time of the first CXL device 31 as a ratio of the entire time (e.g., one hour, one day, one week, one month, etc.). In some example embodiments, the usage ratio may indicate the capacity being used among the entire capacity (memory capacity or storage space) of the first CXL device 31 as a ratio. In some example embodiments, the usage ratio may indicate the access frequency of the corresponding data among the entire data as a ratio. When the usage ratio is greater than a reference value, the first CXL device 31 may move the data.

[0038] In some example embodiments, the first CXL device 31 may determine a CXL device with a usage ratio equal to or less than a reference value as a target device from among the second to Nth CXL devices 32 to 33. The first CXL device 31 may move data to the determined CXL device as the target device.

[0039] In some example embodiments, the first CXL device 31 may determine the data (first data) as the movement target from the stored data based on at least one of the data capacity and / or the access frequency. For example, the first CXL device 31 may determine the data that is accessed the most each time as the first data. The first CXL device 31 may determine the data with the largest capacity as the first data. The first CXL device 31 may move a portion of the first data.

[0040] In some example embodiments, the first CXL device 31 may determine a ratio of the data (second data) to be moved among the first data and the remaining data (third data). The first CXL device 31 may determine a ratio of the second data and the third data based on a data rate ratio of the first CXL device 31 and a target device (e.g., the second CXL device 32). The data rate ratio may represent a ratio of a first data rate between the host and the first CXL device 31 to a second data rate between the host and the second CXL device 32. When the data rate ratio of the first CXL device 31 and the second CXL device 32 is determined to be a:b, the first CXL device 31 may determine a size ratio of the third data and the second data as a:b (a and b are real numbers greater than 0).

[0041] Figure 1 A configuration is shown in which the first CXL device 31 moves the second data to the second CXL device 32, but the embodiment is not limited thereto, and the first CXL device 31 may move a portion of the first data to at least two CXL devices. For example, the first CXL device 31 may move the second data among the first data to the third CXL device, and may move the third data to the fourth CXL device. The first CXL device 31 may store the fourth data in addition to the second data and the third data among the first data. In response to a read request for the first data, the first CXL device 31 may output the fourth data, and the third and fourth CXL devices may output the second and third data, respectively.

[0042] Figures 2 to 6 A block diagram illustrating the operation of a computing system according to some example embodiments, Figure 7 shows an example of a mapping table managed by a first CXL storage device according to some example embodiments, Figure 8 An example of a mapping table managed by a second CXL storage device according to some example embodiments is shown.

[0043] refer to Figure 2 , the computing system 100 may include a host 110, a buffer 111, a CXL switch 120, a first CXL storage device 130 and a second CXL storage device 140. For better understanding and ease of description, Figure 2 The computing system 100 is shown to include one host 110 and two CXL storage devices 130 and 140, but example embodiments are not limited thereto, and the computing system 100 may include at least one host and at least two CXL storage devices. For example, the computing system 100 may include a third CXL device. The computing system 100 may include N CXL devices (N is an integer greater than 0).

[0044] The host 110 may send a command to at least one of the first CXL storage device 130 and the second CXL storage device 140. For example, the host 110 may include a CXL controller, and the CXL controller may communicate with the CXL storage devices 130 and 140 through the CXL switch 120.

[0045] The host 110 may be connected to a buffer 111 storing data. The buffer 111 may be used as a main memory or system memory of the computing system 100. In some example embodiments, the buffer 111 may be a DRAM device and may have a variety of form factors, such as a DIMM or HBM. However, the scope of the present disclosure is not limited thereto, and the buffer 111 may include a non-volatile memory, such as a flash memory, a PRAM, a RRAM, an MRAM, or a FRAM.

[0046] The buffer 111 may communicate directly with the host 110 through a DDR interface. In some example embodiments, the host 110 may include a memory controller for controlling the buffer 111. However, the scope of the present disclosure is not limited thereto, and the buffer 111 may communicate with the host 110 through various types of interfaces. The host 110 may be connected to at least one buffer 111 and may communicate therewith.

[0047] The buffer 111 may store data received from at least one of the first CXL storage device 130 and the second CXL storage device 140. According to some example embodiments, the buffer 111 may be implemented as a memory disposed in the host 110.

[0048] At least one of the first CXL memory device 130 and the second CXL memory device 140 may receive a command from the host 110. The CXL memory devices 130 and 140 may include nonvolatile memories (NVMs) 131 and 141 and controllers 132 and 142. The nonvolatile memories 131 and 141 may store data. The nonvolatile memories 131 and 141 may write, output, or erase data based on the control of the controllers 132 and 142. However, example embodiments are not limited thereto. For example, the computing system 100 may include N nonvolatile memories (N is an integer greater than ).

[0049] The controllers 132 and 142 may control operations of the CXL storage devices 130 and 140 based on an instruction of the host 110. For example, the controllers 132 and 142 may write data to the nonvolatile memories 131 and 141, or may output or erase data of the nonvolatile memories 131 and 141, based on an instruction of the host 110. In some example embodiments, the controllers 132 and 142 may write data to the nonvolatile memories 141 and 131 of the CXL storage devices, or may output or erase data of the nonvolatile memories 141 and 131. For example, the controller 132 of the first CXL storage device 130 may write data to the nonvolatile memory 141 of the second CXL storage device 140, or may output or erase data of the nonvolatile memory 141.

[0050] Controllers 132 and 142 may be connected to CXL switch 120. Controllers 132 and 142 may communicate with host 110 and / or other CXL devices through CXL switch 120. Controllers 132 and 142 may include an architecture for PCIe 5.0 (or other versions) for CXL.io paths, or may add specific CXL.cache and CXL.mem paths to CXL. In some example embodiments, controllers 132 and 142 may be backward compatible with previous cache coherence protocols, such as CXL 1.1 or CXL 2.0. Controllers 132 and 142 may implement CXL.io, CXL.mem, and CXL.cache protocols or other appropriate cache coherence protocols. Controllers 132 and 142 may support different CXL device types, such as type 1, type 2, and / or type 3 CXL devices. Controllers 132 and 142 may support PCIe protocols, such as PCIe 5.0 protocols or PCIe 6.0 protocols. Controllers 132 and 142 may support the Pipeline 5.x protocol by using any and appropriate PHY interface for a PCI Express (PIPE) interface width (eg, 8-bit, 16-bit, 32-bit, 64-bit, and 128-bit configurable PIPE interface widths).

[0051] Controllers 132 and 142 may include intellectual property (IP) circuits designed to implement application specific integrated circuits (ASICs) and / or field programmable gate arrays (FPGAs). In various embodiments, controllers 132 and 142 may support a CXL interface (eg, CXL 3.0 specification or any other version).

[0052] The host 110, the first CXL storage device 130, and the second CXL storage device 140 may communicate with each other through the CXL switch 120. The CXL switch 120 may be included in the CXL interface. The CXL switch 120 may be used to implement a storage cluster through one-to-many and many-to-one switching between the connected CXL storage devices 130 and 140. For example, (i) the CXL switch 120 may connect a routing port to an endpoint, (ii) may connect a routing port to an endpoint, or (iii) may connect a routing port to an endpoint.

[0053] The CXL switch 120 may provide a packet switching function on a CXL packet. The CXL switch 120 may connect the CXL storage devices 130 and 140 to at least one host 110. The CXL switch 120 may (i) allow the CXL storage devices 130 and 140 to include multiple types of memory with different characteristics, (ii) virtualize the memory of the CXL storage devices 130 and 140 and store data with different characteristics (e.g., access frequency) in the appropriate type of memory, and (iii) support remote direct memory access (RDMA). Here, virtualizing the memory may mean converting a memory address between a processing circuit and a memory.

[0054] The CXL switch 120 may arbitrate communications between the host 110, the first CXL storage device 130, and the second CXL storage device 140. For example, when the host 110 communicates with the CXL storage devices 130 and 140, the CXL switch 120 may transmit information such as a request, data, response, or signal transmitted from the host 110 or the CXL storage devices 130 and 140 to the CXL storage devices 130 and 140 or the host 110 through the CXL.io protocol.

[0055] When the first CXL storage device 130 communicates with the second CXL storage device 140 , the CXL switch 120 may transmit information such as a request, data, a response, or a signal between the first CXL storage device 130 and the second CXL storage device 140 through the CXL.mem protocol.

[0056] The first CXL storage device 130 communicates with the second CXL storage device 140 through the CXL.mem protocol, and they may operate independently of the CXL.io protocol of the host 110. In other words, the controller 132 may communicate with the host 110 through the CXL.io protocol, and may communicate with the controller 142 through the CXL.mem protocol. Therefore, the communication between the first CXL storage device 130 and the second CXL storage device 140 may have limited impact on the input / output latency of the host 110.

[0057] refer to Figure 3 , the host 110 may send an instruction to the first CXL storage device 130 through the CXL.io protocol of the CXL switch 120. For example, the instruction may be a read request for the first data DAT1. The host 110 may send the address of the first data DAT1 and the address of the buffer 111 together with the read request for the first data DAT1 to the first CXL storage device 130. The first CXL storage device 130 may receive the read request for the first data DAT1, the address of the first data DAT1, and the address of the buffer 111 from the host 110.

[0058] refer to Figure 4 , the first CXL storage device 130 may read the first data DAT1 from the nonvolatile memory 131 based on the address of the first data DAT1. The first CXL storage device 130 may write the first data DAT1 to the buffer 111 based on the address of the buffer 111. The first CXL storage device 130 may transmit the first data DAT1 through the CXL.io protocol of the CXL switch 120. The buffer 111 may receive the first data DAT1 from the first CXL storage device 130 and may store them.

[0059] refer to Figure 5 , the first CXL storage device 130 may calculate the usage ratio. For example, Figure 2 The controller 132 may calculate a usage ratio of the first CXL storage device 130. In some example embodiments, the usage ratio may indicate the operation time of the first CXL storage device 130 as a ratio of the entire time (e.g., one hour, one day, one week, one month, etc.). The operation time may represent a time when the power is turned off or a non-idle state is maintained. In some example embodiments, the usage ratio may be a ratio of the capacity being used among the entire capacity of the nonvolatile memory 131. In some example embodiments, the usage ratio may be a ratio of the access frequency of all data stored in the nonvolatile memory 131. When the usage ratio is greater than a reference value, the controller 132 may determine to move the data to another CXL device.

[0060] The controller 132 may determine a target device to which the data is to be moved. The controller 132 may obtain a usage ratio of each CXL device connected through the CXL switch 120 .

[0061] The controller 132 may determine at least one of the CXL devices as a target device based on the usage ratios of the respective CXL devices. For example, the controller 132 may determine the second CXL storage device 140 having the lowest usage ratio among the CXL devices as a target device. In some example embodiments, the controller 132 may determine another CXL device as a target device in addition to the second CXL storage device 140.

[0062] The controller 132 may determine the target data to be moved. The controller 132 may determine the number of read requests for data stored in the nonvolatile memory 131. In some example embodiments, the controller 132 may determine the number of read requests within a reference time. The controller 132 may determine whether the number of read requests is greater than a reference number.

[0063] The controller 132 may determine the target data based on the read request number. In some example embodiments, the controller 132 may determine some data whose read request number is greater than the reference number from among the data of the nonvolatile memory 131 as the target data. In some example embodiments, the controller 132 may determine some data whose read request number is the largest from among the data of the nonvolatile memory 131 as the target data. For example, the controller 132 may determine the second data DAT2 which is a part of the first data DAT1 as the target data. That is, the controller 132 may determine that the second data DAT2 is stored in the second CXL storage device 140. The controller 132 may output the second data DAT2 to the second CXL storage device 140.

[0064] In some example embodiments, the controller 132 may determine the target data. For example, the controller 132 may determine some data with the second largest number of read requests as the target data. The controller 132 may determine the size ratio among the target data.

[0065] The controller 132 may determine the size of the target data. That is, the controller 132 may determine the ratio of the second data DAT2 to the first data DAT1. For example, the controller 132 may determine the size of the target data based on the rate of the first CXL storage device 130, the rate of the target device, and the size of the target data. The first data DAT1 may be sequential data, and the second data DAT2 may be the end portion of the first data DAT1. For example, when the controller 132 has determined that the size of the target data is 10%, the controller 132 may determine the last 10% of the first data DAT1 as the second data DAT2.

[0066] In some example embodiments, the controller 132 may obtain a first transmission rate between the first CXL storage device 130 and the host 110. The first transmission rate may represent a data communication rate of the first CXL storage device 130 through the CXL.io protocol. The controller 132 may determine a ratio of the first data DAT1 and the second data DAT2 as target data based on the first transmission rate. For example, when the first transmission rate is high, the controller 132 may determine the ratio of the second data DAT2 in a relatively low manner. In a similar manner, when the first transmission rate is low, the controller 132 may determine the ratio of the second data DAT2 in a relatively high manner.

[0067] In some example embodiments, the controller 132 may obtain a second transmission rate between the host 110 and the second CXL storage device 140. The second transmission rate may represent a data communication rate of the second CXL storage device 140 through the CXL.io protocol. The controller 132 may determine the ratio based on the first transmission rate and the second transmission rate. The controller 132 may determine the ratio of the first data DAT1 and the second data DAT2 such that “(first transmission rate) / (first transmission rate+second transmission rate)” corresponds to “size of the second data DAT2 / size of the first data DAT1”. For example, the controller 132 may determine the ratio of the first data DAT1 and the second data DAT2 such that “(first transmission rate) / (first transmission rate+second transmission rate)” is proportional to or equal to “size of the second data DAT2 / size of the first data DAT1”. In other words, when a ratio (e.g., a ratio of the first transmission rate to the target transmission rate (e.g., the second transmission rate) is a:b, the controller 132 may determine a size ratio of the remaining data to the target data (e.g., DAT2) as a:b. For example, in some example embodiments where the computing system 100 includes N (N is an integer greater than 1) CXL storage devices, when a ratio of the first transmission rate to the target transmission rate (e.g., a transmission rate between the host and the Nth CXL storage device) is a:b, the controller 132 may determine a size ratio of the remaining data to the Nth data (e.g., data to be moved to the Nth CXL storage device) as a:b.

[0068] The controller 132 may determine whether to send the target data to the target device. In some example embodiments, the controller 132 may obtain a first time used when the first CXL storage device 130 sends the first data DAT1 to the host 110, and may obtain a second transmission rate between the host 110 and the second CXL storage device 140. The controller 132 may determine whether to output the second data DAT2 based on the first time, the second transmission rate, and the size of the second data DAT2. For example, the controller 132 may calculate the second time of the second CXL storage device 140 based on the second transmission rate and the size of the second data DAT2. When the first time is longer than the second time, the controller 132 may output the second data DAT2 to the second CXL storage device 140. When the first time is equal to or shorter than the second time, the controller 132 may not output the second data DAT2 to the second CXL storage device 140. For example, in some example embodiments where the computing system 100 includes N CXL storage devices, the controller 132 may calculate the Nth time based on the Nth transmission rate and the size of the Nth data. When the first time is longer than the Nth time, the controller may output the Nth data to the Nth CXL memory device.

[0069] In some example embodiments, the controller 132 may obtain a first time used when transmitting the first data DAT1 to the host 110, may obtain a first transmission rate between the first CXL storage device 130 and the host 110, and may obtain a second transmission rate between the host 110 and the second CXL storage device 140. The controller 132 may determine whether to output the second data DAT2 based on the first time, the first data DAT1, the second data DAT2, the first transmission rate, and the second transmission rate. For example, the controller 132 may calculate a second time to transmit the third data DAT3 based on the size of the third data DAT3 excluding the second data DAT2 among the first data DAT1 and the first transmission rate. The controller 132 may calculate a third time to transmit the second data DAT2 based on the second transmission rate and the size of the second data DAT2. When the first time is longer than the sum of the second time and the third time, the controller 132 may output the second data DAT2 to the second CXL storage device 140. When the first time is equal to or shorter than the sum of the second time and the third time, the controller 132 may not output the second data DAT2 to the second CXL storage device 140. For example, in some example embodiments where the computing system 100 includes N CXL storage devices, the controller 132 may obtain the transmission rates of the first to Nth CXL storage devices and the sizes of the data corresponding to the first to Nth CXL storage devices. The controller 132 may calculate the lengths of the first to Nth times based on the corresponding first to Nth transmission rates and the first to Nth data sizes. When the first time is longer than the sum of the second to Nth times, the controller may output the second to Nth data to the corresponding second to Nth CXL storage devices.

[0070] The controller 132 may send the target data to the target device based on the target device, the target data, the size of the target data, and the determination of the transmission. For example, the controller 132 may send the second data DAT2 to the second CXL storage device 140 by using the CXL.mem protocol. Figure 2 The controller 142 may store the data received from the first CXL storage device 130 in the nonvolatile memory 141 .

[0071] In some example embodiments, the controller 132 may indicate a first address to the controller 142, the first address being a location where the second data DAT2 is stored. The controller 142 may store the second data DAT2 in the nonvolatile memory 141 based on the first address. The controller 132 may write the first address to the mapping table. When other data is written to the first address, the controller 142 may write the second data DAT2 to the second address. The controller 142 may send the second address to the controller 132, and the controller 132 may write the second address to the mapping table.

[0072] In some example embodiments, the controller 132 may not indicate the location where the second data DAT2 is stored to the controller 142. The controller 142 may determine that the second data DAT2 is stored at a third address in the nonvolatile memory 141. The controller 142 may store the second data DAT2 at the third address and may send the third address to the controller 132. The controller 132 may write the third address to the mapping table.

[0073] The controller 132 may determine whether the second CXL storage device 140 has completed storage of the second data DAT2. In some example embodiments, when storing the second data DAT2 in the nonvolatile memory 141, the controller 142 of the second CXL storage device 140 may send a storage completion signal to the controller 132. When receiving the storage completion signal from the second CXL storage device 140, the controller 132 may determine that storage of the second data DAT2 has been completed.

[0074] In some example embodiments, when storage of the second data DAT2 has been completed, the second CXL storage device 140 may change a bit value of a region of the nonvolatile memory 141, and the controller 132 may determine whether storage of the second data DAT2 by the second CXL storage device 140 is completed based on the bit value of the region. For example, when the bit value of the region changes, the controller 132 may check whether storage of the second data DAT2 is completed, and when the bit value of the region does not change, the controller 132 may check whether storage of the second data DAT2 is completed.

[0075] refer to Figure 6 , the first CXL storage device 130 may move the second data DAT2 among the first data DAT1 to the second CXL storage device 140, and may store the third data DAT3 among the first data DAT1 in the nonvolatile memory 131. The second CXL storage device 140 may store the second data DAT2 in the nonvolatile memory 141. That is, when the usage rate of the first CXL storage device 130 is high, the second data DAT2, which is a part of the first data DAT1 of the first CXL storage device 130, is dispersed to the second CXL storage device 140, thereby reducing the waiting time of the first data DAT1 of the host (including 110) of the computing system 100 and alleviating or reducing the data skew.

[0076] refer to Figures 6 to 8 , the first CXL storage device 130 may manage the mapping table 1300 , and the second CXL storage device 140 may manage the mapping table 1400 .

[0077] Before moving the second data DAT2, the first CXL storage device 130 may have stored the address of the first data DAT1 in the areas 1310 and 1320 of the mapping table 1300. The first CXL storage device 130 may move the second data DAT2 to the second CXL storage device 140, and may correct (or update) the mapping table 1300. The first CXL storage device 130 may correct the address of the second data DAT2 on the mapping table 1300.

[0078] For example, the first CXL storage device 130 may hold addresses P1 to P9 of the third data DAT3 among the first data DAT1. The first CXL storage device 130 may correct the address of the second data DAT2 from among the first data DAT1. That is, the first CXL storage device 130 may hold addresses P1 to P9 of the region 1310, and may correct the address of the region 1320. The sizes of the region 1310 and the region 1320 may be proportional to the sizes of the first data DAT1 and the second data DAT2, respectively. That is, when the size of the second data DAT2 moved to the second CXL storage device 140 increases, the size of the region 1310 may decrease, and the size of the region 1320 may increase.

[0079] In some example embodiments, the first CXL storage device 130 may indicate an address where the second data DAT2 is stored to the second CXL storage device 140. The address where the second data DAT2 is stored may be an address of a nonvolatile memory 141 of the second CXL storage device 140. In some example embodiments, the first CXL storage device 130 may not indicate an address where the second data DAT2 is stored to the second CXL storage device 140. The second CXL storage device 140 may write the second data DAT2 to the nonvolatile memory 141, and may write the address P10_CD1 where the second data DAT2 is written to the area 1410 of the mapping table 1400.

[0080] The second CXL storage device 140 may send information P_CD2 on region 1410 to the first CXL storage device 130. The first CXL storage device 130 may write the information P_CD2 to region 1320 of the mapping table 1300. The information P_CD2 may correspond to address P10_CD1. In some example embodiments, the information P_CD2 may be address P10_CD1.

[0081] When receiving a read request for first data DAT1 from the host 110, the first CXL storage device 130 may request the second CXL storage device 140 to transmit second data DAT2 based on the information P_CD2 of the area 1320. The second CXL storage device 140 may transmit the second data DAT2 to the host 110 in response to the request of the first CXL storage device 130.

[0082] Figures 9 to 12 A block diagram illustrating the operation of a computing system according to an embodiment is shown.

[0083] refer to Fig. 9 , computing system 100 may include host 110, buffer 111, CXL switch 120, and CXL storage devices 130 and 140. Figures 2 to 8 The contents described are applicable to host 110, buffer 111, CXL switch 120, and CXL storage devices 130 and 140. Repeated contents thereof will not be provided.

[0084] The first CXL memory device 130 may store the third data DAT3 in the nonvolatile memory 131 , and the second CXL memory device 140 may store the second data DAT2 in the nonvolatile memory 141 .

[0085] The host 110 may send a read request on the first data DAT1 to the first CXL storage device 130 through the CXL.io protocol. The host 110 may send a buffer address PSA1 and an address of the first data DAT1 together with the read request. The buffer address PSA1 may be an address for writing the read first data DAT1. The address of the first data DAT1 may indicate a location in the nonvolatile memory 131 where the first data DAT1 is written. The first CXL storage device 130 may read the third data DAT3 from the nonvolatile memory 131 based on the address of the first data DAT1. The first CXL storage device 130 may write the third data DAT3 to the buffer 111 based on the buffer address PSA1.

[0086] When receiving a read request for the first data DAT1 from the host 110, the controller 132 may determine whether the second CXL storage device 140 has completed storing the second data DAT2. The controller 132 may operate differently based on whether the second CXL storage device 140 has completed storing.

[0087] In some example embodiments, when the second CXL storage device 140 has not completed storing the second data DAT2, the controller 132 may Figure 4The controller 132 may transmit the first data DAT1 to the host 110. In some example embodiments, when the second CXL storage device 140 has not completed storing the second data DAT2, the controller 132 may wait until the second CXL storage device 140 completes storing the second data DAT2.

[0088] Fig.10 The exemplary embodiment shows a case where the second CXL storage device 140 has completed storing the second data DAT2. Fig.10 , when receiving a read request for the first data DAT1 from the host 110, the controller 132 may send the third data DAT3 except the second data DAT2 among the first data DAT1 to the host 110. The controller 132 may read the third data DAT3 based on the mapping table (eg, Figure 7 The controller 132 may read the third data DAT3 from the nonvolatile memory 131 at an address (eg, P1 to P9) of the nonvolatile memory 131. The controller 132 may send the third data DAT3 to the host 110 using the CXL.io protocol.

[0089] Furthermore, when the second CXL storage device 140 finishes storing the second data DAT2 , the controller 132 may request the second CXL storage device 140 to transmit the second data DAT2 to the host 110 .

[0090] The controller 132 may request the second CXL memory device 140 to transmit the second data DAT2 to the host 110. The controller 132 may request the second CXL memory device 140 to transmit the second data DAT2 using the CXL.mem protocol.

[0091] The controller 132 may be configured based on a mapping table (eg, Figure 7 1300) in the address (for example, Figure 7 The controller 132 may generate a request signal for the second data DAT2 by using P_CD2 in the memory device 140. In some example embodiments, the request signal may include an address. The controller 132 may send the request signal for the second data DAT2 to the second CXL memory device 140.

[0092] The controller 132 may determine the buffer address PSA2 at which the second CXL storage device 140 writes the second data DAT2 based on the buffer address PSA1 and the mapping table. For example, the controller 132 may obtain the length of the third data DAT3 on the mapping table, may add the length of the third data DAT3 to the buffer address PSA1, and may determine the buffer address PSA2. In some example embodiments, the controller 132 may determine the length of the third data DAT3 based on a difference between a start point and an end point of the third data DAT3 on the mapping table. The controller 132 may indicate the buffer address PSA2 to the second CXL storage device 140.

[0093] refer to Fig.11 , the second CXL storage device 140 may transmit the second data DAT2 to the host 110 by using the CXL.io protocol in response to the transmission request of the controller 132. The second CXL storage device 140 may transmit the second data DAT2 to the buffer 111 based on the buffer address PSA2.

[0094] The first CXL memory device 130 may transmit the third data DAT3 from the buffering address PSA1 to the buffering address PSA2 .

[0095] refer to Fig.12 , the controller 132 may determine whether the controller 142 has completed the transmission of the second data DAT2 to the buffer 111. The controller 132 may check the completion of the transmission through the CXL.mem protocol. In some example embodiments, when the transmission of the second data DAT2 has been completed, the controller 142 may send a transmission completion signal to the controller 132. In some example embodiments, when the transmission of the second data DAT2 has been completed, the controller 142 may change the status bit. The controller 132 may check the change of the status bit, and may check that the second CXL storage device 140 has completed the transmission of the second data DAT2.

[0096] When the controller 132 has completed transmission of the third data DAT3 and the controller 142 has completed transmission of the second data DAT2, they may transmit a transmission completion signal of the first data DAT1 to the host 110. The controller 132 may transmit the transmission completion signal of the first data DAT1 through the CXL.io protocol.

[0097] As described above, in response to a read request for the first data DAT1, the first CXL memory device 130 may send the third data DAT3 and the second CXL memory device 140 may send the second data DAT2, thereby reducing latency on the first data DAT1 of the host (including 110) of the computing system 100 and alleviating or reducing data skew.

[0098] Fig.13 A block diagram of a computing system is shown, according to some example embodiments.

[0099] refer to Fig.13 , computing system 200 may include host 210, buffer 211, CXL switch 220, first CXL memory device 230 and second CXL memory device 240. For better understanding and ease of description, Fig.13It is shown that the computing system 200 includes one host 210 and two CXL memory devices 230 and 240 , example embodiments are not limited thereto, and the computing system 200 may include at least one host and at least two CXL memory devices.

[0100] The host 210 may transmit the command to at least one of the first CXL memory device 230 and the second CXL memory device 240. For example, the host 210 may include a CXL controller, and the CXL controller may communicate with the CXL memory devices 230 and 240 through the CXL switch 220.

[0101] The host 210 may be connected to a buffer 211 storing data. The buffer 211 may be used as a main memory or a system memory of the computing system 200. In some example embodiments, the buffer 211 may have a variety of form factors, such as a DRAM device, a DIMM, or an HBM. However, the scope of the present disclosure is not limited thereto, and the buffer 211 may include a non-volatile memory, such as a flash memory, a PRAM, an RRAM, an MRAM, or a FRAM.

[0102] The buffer 211 may communicate with the host 210 through a DDR interface. In some example embodiments, the host 210 may include a memory controller for controlling the buffer 211. However, the scope of the present disclosure is not limited thereto, and the buffer 211 may communicate with the host 210 through various types of interfaces. The host 210 may be connected to at least one buffer 211 and may communicate therewith.

[0103] The buffer 211 may store data received from at least one of the first CXL memory device 230 and the second CXL memory device 240. According to this embodiment, the buffer 211 may be implemented as a memory provided in the host 210.

[0104] At least one of the first CXL memory device 230 and the second CXL memory device 240 may receive a command from the host 210. The CXL memory devices 230 and 240 may include a controller and a memory cell array. The memory cell array may write, output, or erase data based on the control of the controller. The CXL memory devices 230 and 240 may be volatile memories.

[0105] The host 210, the first CXL memory device 230, and the second CXL memory device 240 may communicate with each other through the CXL switch 220. The CXL switch 220 may be included in a CXL interface.

[0106] The CXL switch 220 may arbitrate communications between the host 210, the first CXL memory device 230, and the second CXL memory device 240. For example, when the host 210 communicates with the CXL memory devices 230 and 240, the CXL switch 220 may transmit information such as a request, data, response, or signal transmitted from the host 210 or the CXL memory devices 230 and 240 to the CXL memory devices 230 and 240 or the host 210 through the CXL.io protocol.

[0107] When the first CXL memory device 230 communicates with the second CXL memory device 240 , the CXL switch 220 may transmit information such as a request, data, a response, or a signal between the first CXL memory device 230 and the second CXL memory device 240 through the CXL.mem protocol.

[0108] The first CXL memory device 230 and the second CXL memory device 240 may communicate with each other via the CXL.mem protocol, and may operate independently of the CXL.io protocol of the host 210. In other words, the first CXL memory device 230 may communicate with the host 210 via the CXL.io protocol, and may communicate with the second CXL memory device 240 via the CXL.mem protocol. Therefore, the communication between the first CXL memory device 230 and the second CXL memory device 240 may have limited impact on the input / output latency of the host 210.

[0109] The first CXL memory device 230 may determine a usage ratio. When the usage ratio is greater than a reference value, the first CXL memory device 230 may move data stored in the memory.

[0110] The first CXL memory device 230 may determine the target data to be moved. In some example embodiments, the first CXL memory device 230 may determine the target data based on the read request number. For example, the first CXL memory device 230 may determine the second data that is a part of the first data having the largest read request number as the target data.

[0111] The first CXL memory device 230 may determine a target device for moving target data. In some example embodiments, the first CXL memory device 230 may determine at least one of the CXL devices (including the second CXL memory device 240) sharing memory through the CXL switch 220 as the target device. The first CXL memory device 230 may determine the second CXL memory device 240 having the lowest usage ratio from among the CXL devices as the target device.

[0112] The first CXL memory device 230 may determine the size of the target data. The first CXL memory device 230 may determine the size of the target data based on a first transfer rate between the first CXL memory device 230 and the host 210 and a second transfer rate between the second CXL memory device 240 and the host 210. For example, the first CXL memory device 230 may determine the size of the second data so that a ratio of a size occupied by the second data in the first data having the largest number of read requests becomes “second transfer rate / (first transfer rate+second transfer rate)”.

[0113] The first CXL memory device 230 may send the determined target data to the determined target device (the second CXL memory device 240). The second CXL memory device 240 may store the target data received from the first CXL memory device 230. When the host 210 sends a read request for the first data to the first CXL memory device 230, the first CXL memory device 230 may send the third data in the first data except the second data to the host 210, and the second CXL memory device 240 may send the second data to the host 210. As described above, when the usage rate of the first CXL memory device 230 is high, the second data, which is a part of the first data of the first CXL memory device 230, is dispersed to the second CXL memory device 240, thereby reducing the latency of the first data of the host (including 210) of the computing system 200, and alleviating or reducing the data skew.

[0114] Fig.14 A block diagram of a computing system is shown according to one embodiment.

[0115] refer to Fig.14 , computing system 300 may include host 310, buffer 311, CXL switch 320, CXL memory device 330 and CXL storage device 340. For better understanding and ease of description, Fig.13 The computing system 300 is shown to include one host 310 , one CXL memory device 330 , and one CXL storage device 340 , but the embodiment is not limited thereto, and the computing system 300 may include one or more hosts, one or more CXL memories, and one or more CXL storage devices.

[0116] right Figure 2 The description of host 110, buffer 111, CXL switch 120, and CXL storage devices 130 and 140 may be applied to Fig.14 The host 310, the buffer 311, the CXL switch 320 and the CXL storage device 340. Fig.13 The description of CXL memory devices 230 and 240 may be applied to Fig.14The CXL memory device 330 will not provide its duplicate content.

[0117] That is, when the usage rate of the CXL memory device 330 is high, the second data, which is a part of the first data with high usage frequency (with a large number of read requests) in the CXL memory device 330, is dispersed to the CXL memory device 340, thereby reducing the waiting time of the first data of the host (including 310) of the computing system 300 and alleviating or reducing data skew. In a similar manner, when the usage rate of the CXL memory device 340 is high, a part of the data with high usage frequency of the CXL memory device 340 may be dispersed to the CXL memory device 330.

[0118] Fig.15 A flow chart showing a method of storing data between CXL devices according to some example embodiments. Fig.15 A case is shown in which a CXL device (eg, a first CXL device (CXL DEVICE 1)) moves data to another CXL device (eg, a second CXL device (CXL DEVICE 2)) to store the data.

[0119] refer to Fig.15 , the host may send a command (RQ_IO) for reading the first data DAT1 to the first CXL device through the CXL switch (CXL SWITCH) (S1510). The host may send the command (RQ_IO) through the CXL.io protocol. The first CXL device may send the stored first data DAT1 to the host in response to the command (RQ_IO). The first CXL device may send the first data DAT1 to a buffer memory connected to the host through the CXL.io protocol.

[0120] The first CXL device may determine another CXL device based on the usage record (S1520). For example, the first CXL device may determine a usage ratio based on the usage record. The usage ratio may indicate the operation time of the first CXL device using a ratio per unit time (e.g., one hour, one day, one week, one month, etc.).

[0121] When the usage ratio is greater than the reference value, the first CXL device may determine another CXL device. The first CXL device may determine at least one of the other CXL devices connected through the CXL switch as a target device. The target device may represent a device for moving data stored in the first CXL device. The first CXL device may determine a CXL device having a usage ratio equal to or less than the reference value from among the CXL devices as the target device. In some example embodiments, the first CXL device may determine a CXL device having a lowest usage ratio from among the CXL devices as the target device.

[0122] The first CXL device may determine the target data based on the read request number. For example, when the number of read requests (e.g., the number of received commands (RQ_IO)) of the first data DAT1 is the largest among the data stored in the first CXL device, the second data DAT2 which is at least a part of the first data DAT1 may be determined as the target data. The second data DAT2 may correspond to the end portion of the first data DAT1.

[0123] The first CXL device may send a command (RQ_PGM) for writing the second data DAT2 to the second CXL device as the target device (S1530). The first CXL device may send the command (RQ_PGM) through the CXL.mem protocol. The second CXL device may write the second data DAT2 in response to the command (RQ_PGM). When the writing of the second data DAT2 is completed, the second CXL device may send a completion signal (ACK_PGM) to the first CXL device. The second CXL device may send the completion signal (ACK_PGM) through the CXL.mem protocol. In some example embodiments, the first CXL device may send an address for writing the second data DAT2 to the second CXL device. In some example embodiments, the second CXL device may write the second data DAT2, and may send the address of the second data DAT2 that has been written to the first CXL device. The first CXL device may write the address of the second data DAT2 into the mapping table.

[0124] Fig.16 A flow chart showing a data transmission method of a CXL device according to some example embodiments is shown. Fig.16 It is shown that the first CXL device and the second CXL device send data to the host in response to a read request of the host.

[0125] refer to Fig.16 , the first CXL device may store third data DAT3 (S1610). The third data DAT3 may represent Fig.15 The data of the first data DAT1 except the second data DAT2.

[0126] The second CXL device may hold the second data DAT2 (S1620). The second data DAT2 may correspond to an end portion of the first data DAT1.

[0127] The host may send a command (RQ_IO) for reading the first data DAT1 to the first CXL device (S1630). The host may send the command (RQ_IO) through the CXL.io protocol. The first CXL device may check the mapping data in response to the command (RQ_IO). The first CXL device may read the third data DAT3 based on the mapping data, and may send the third data DAT3 to the host through the CXL switch. The first CXL device may send the third data DAT3 to a buffer memory connected to the host through the CXL.io protocol.

[0128] In addition, the first CXL device may send a command (RQ_DAT2) for reading the second data DAT2 to the second CXL device based on the mapping data. The first CXL device may send the command (RQ_DAT2) to the second CXL device via the CXL switch. The host may send the command (RQ_DAT2) via the CXL.mem protocol. Fig.16 It is shown that the transmission of the command (RQ_DAT2) precedes the transmission of the third data DAT3, but the embodiment is not limited thereto, and the transmission of the command (RQ_DAT2) and the transmission of the third data DAT3 may be generated simultaneously, or the transmission of the command (RQ_DAT2) may be after the transmission of the third data DAT3.

[0129] The second CXL device may send the second data DAT2 to the host through the CXL switch. The second CXL device may send the second data DAT2 to a buffer memory connected to the host through the CXL.io protocol.

[0130] When the transmission of the second data DAT2 is completed, the second CXL device may send a completion signal (ACK_DAT2) to the first CXL device. In an embodiment, the first CXL device may check the completion of the transmission of the second data DAT2 by the second CXL device by checking a bit of a region of the second CXL device.

[0131] Upon receiving the completion signal (ACK_DAT2) from the second CXL device, the first CXL device may send a completion signal (ACK_IO) on the instruction (RQ_IO) to the host through the CXL switch.

[0132] Fig.17 A flow chart of a data storage method according to some example embodiments is shown.

[0133] refer to Fig.17 The data storage method may be performed by a CXL device. The CXL device is connected to a host through a CXL switch and may be implemented by a CXL memory device, a CXL storage device, and the like.

[0134] The CXL device may receive a first read request for first data DATA1 from the host (S1710). The CXL device may send the first data DATA1 to the host. The host and the CXL device may send and receive the read request and the first data DATA1 to each other through the CXL switch.

[0135] When the number of received first read requests is greater than the reference number, the CXL device may select one of the shared memories (S1720). The shared memories may be connected by the CXL devices through a CXL switch.

[0136] The CXL device may determine second data DATA2 to be transmitted to the selected shared memory (S1730). The CXL device may determine the second data DATA2 based on the size of the first data DATA1, the first transmission rate of the selected shared memory, and the second transmission rate of the CXL device. The first transmission rate and the second transmission rate may represent the rates between the device and the host, respectively. The CXL device may determine an end portion of the first data DATA1 having a predetermined or optional expected size. The CXL device may determine the predetermined or optional expected size based on the first transmission rate and the second transmission rate.

[0137] The CXL device may write the second data DATA2 into the selected shared memory (S1740). The CXL device may send the second data DATA2 to the selected shared memory by using the CXL.mem protocol.

[0138] When receiving a second read request for the first data DATA1 from the host, the CXL device may request the selected shared memory for transmission of the second data DATA2, and may send third data DATA3 except the second data DATA2 among the first data DATA1 to the host (S1750). The CXL device may send the third data DATA3 to the host through the CXL.io protocol, and may send the transmission of the second data DATA2 to the shared memory through the CXL.mem protocol.

[0139] Fig.18 A block diagram of a computing system is shown, according to some example embodiments.

[0140] refer to Fig.18, computing system 1800 may include host 1810, memory devices 1811 and 1812, CXL storage 1820, and CXL memory 1830. In some example embodiments, computing system 1800 may be included in a user device such as a personal computer, a laptop, a server, a data center, a media player, or a digital camera, or in an automotive device such as a GPS, a black box, or a vehicle electronic device. Alternatively, computing system 1800 may be a mobile system such as a mobile phone, a smart phone, a tablet personal computer, a wearable device, a health care device, or an Internet of Things (IOT) device. In one embodiment, computing system 1800 may provide artificial intelligence (AI) services.

[0141] The host 1810 may control the general operation of the computing system 1800. In one embodiment, the host 1810 may be one of a variety of processors, such as a CPU, a GPU, an NPU, or a DPU. In some example embodiments, the host 1810 may include a single-core processor or a multi-core processor.

[0142] The memory devices 1811 and 1812 may be used as a main memory or a system memory of the computing system 1800. In some example embodiments, the memory devices 1811 and 1812 may be DRAM devices, respectively, and may have a variety of form factors, such as DIMM or HBM. However, the scope of the present disclosure is not limited thereto, and the memory devices 1811 and 1812 may include non-volatile memory, such as flash memory, PRAM, RRAM, MRAM, or FRAM.

[0143] The memory devices 1811 and 1812 may communicate directly with the host 1810 through a DDR interface. In some example embodiments, the host 1810 may include a memory controller for controlling the memory devices 1811 and 1812. However, the scope of the present disclosure is not limited thereto, and the memory devices 1811 and 1812 may communicate with the host 1810 through various types of interfaces.

[0144] The CXL storage 1820 may include a CXL storage controller 1821 and a nonvolatile memory NVM. According to the control of the host 1810, the CXL storage controller 1821 may store data in the nonvolatile memory NVM, or may transmit the data stored in the nonvolatile memory NVM to the host 1810 or the CXL storage 1830. The nonvolatile memory NVM may store data, and may retain the data when the CXL storage 1820 is turned off. In some example embodiments, the nonvolatile memory NVM may be a NAND flash memory, and the scope of the present disclosure is not limited thereto.

[0145] The CXL memory 1830 may include a CXL memory controller 1831 and a buffer memory BFM. The CXL memory controller 1831 may store data in the buffer memory BFM or may transmit data stored in the buffer memory BFM to the host 1810 according to the control of the host 1810. For example, the CXL memory controller 1831 may store data of the memory devices 1811 and 1812 or the CXL memory 1820 in the buffer memory BFM according to the control of the host 1810. In some example embodiments, the buffer memory BFM may be a DRAM, and the scope of the present disclosure is not limited thereto.

[0146] In some example embodiments, the host 1810, the CXL storage 1820, and the CXL memory 1830 may have the same interface. For example, the host 1810, the CXL storage 1820, and the CXL memory 1830 may communicate with each other via a CXL interface 1850. In some example embodiments, the CXL interface 1850 may indicate a low-latency and high-bandwidth link for implementing a variety of connections between accelerators, memory devices, or a variety of electronic devices by supporting consistency of input / output protocols, memory access, and dynamic protocol multiplexing.

[0147] In some example embodiments, the CXL storage 1820 may access the CXL memory 1830 through the CXL interface 1850. For example, the CXL storage 1820 may store data in an allocated area of ​​the CXL memory 1830, or may read the stored data. According to the control of the CXL storage 1820, the CXL memory 1830 may store data in the buffer memory BFM, or may send the data stored in the buffer memory BFM to the CXL storage 1820.

[0148] The CXL storage controller 1821 of the CXL storage 1820 may communicate with the host 1810 and the CXL memory 1830 (e.g., a buffer memory) through the CXL interface 1850. In other words, the CXL storage controller 1821 of the CXL storage 1820 may communicate with the host 1810 and the CXL memory 1830 through the same kind of interface or a common interface, and may use a predetermined or selectable desired area of ​​the CXL memory 1830 as a buffer memory.

[0149] refer to Figures 2 to 12The example embodiments described with respect to the CXL memory devices 130 and 140 may be applied to the CXL memory 1820. That is, when the use ratio is greater than a reference value, the CXL memory controller 1821 may transmit target data to the CXL memory 1830 connected through the CXL interface 1850. The target data may represent second data that is a part of the first data. When a read request for the first data is received from the host 1810, the CXL memory controller 1821 may transmit third data from the nonvolatile memory NVM to the memory device 1811 or 1812. The third data may represent data other than the second data among the first data. The CXL memory controller 1821 may request the CXL memory 1830 to transmit the second data. In response to the request of the CXL memory controller 1821, the CXL memory controller 1831 may transmit the second data from the buffer memory BFM to the memory device 1811 or 1812.

[0150] In some example embodiments, the host 1810, the CXL storage 1820, and the CXL memory 1830 are shown as communicating with each other through the CXL interface 1850, but the scope is not limited thereto, and the host 1810, the CXL storage 1820, and the CXL memory 1830 may communicate with each other based on various types of computing interfaces, such as the GEN-Z protocol, the NVLink protocol, the CCIX protocol, and the open CAPI protocol.

[0151] Furthermore, in the present embodiment, the CXL storage 1820 is described as using the CXL memory 1830 , but example embodiments are not limited thereto, and the CXL memory 1830 may use the CXL storage 1820 .

[0152] Fig.19 A block diagram of a computing system according to some example embodiments is shown. For better understanding and ease of description, the detailed contents of the above-mentioned constituent elements will not be provided.

[0153] refer to Fig.19 , the computing system 1900 may include a CXL switch 1905 , a first CPU 1910 , a second CPU 1920 , a GPU 1930 , an NPU 1940 , a CXL storage 1950 , a CXL memory 1960 , a PCIe device 1970 , and an accelerator (or CXL device) 1980 .

[0154] The first CPU 1910 , the second CPU 1920 , the GPU 1930 , the NPU 1940 , the CXL storage 1950 , the CXL memory 1960 , the PCIe device 1970 , and the accelerator (or CXL device) 1980 may be commonly connected to the CXL switch 1905 , and may communicate with each other through the CXL switch 1905 .

[0155] In some example embodiments, the first CPU 1910, the second CPU 1920, the GPU 1930, and the NPU 1940 may be reference 1910, 1920, 1930, and 1940, respectively. Figures 1 to 18 The host described above may be connected to the memory devices 1911 to 1918, respectively. In some example embodiments, the CXL storage 1950 may be a reference Figures 2 to 12 In some example embodiments, CXL memory 1960 may be a reference to CXL memory devices 130 and 140. Fig.13 and Fig.18 The CXL memory devices 230, 240, and 1830 described above are used to store data. For example, when the usage ratio is greater than a reference value, the CXL storage 1950 or the CXL memory 1960 may determine other CXL devices (e.g., 1950, 1960, and 1970) as target devices. The CXL storage 1950 or the CXL memory 1960 may move data to the target device and may store the data. Therefore, a data bottleneck phenomenon according to a read request of a host may be alleviated or reduced, and data skew caused by the data bottleneck phenomenon may be alleviated or reduced.

[0156] At least one of the first CPU 1910 , the second CPU 1920 , the GPU 1930 , and the NPU 1940 may allocate at least a predetermined or optionally desired area of ​​the CXL memory 1960 as a dedicated area of ​​the CXL storage 1950 .

[0157] That is, CXL storage 1950 and CXL memory 1960 may be used as a storage space STR of computing system 1900 .

[0158] In some example embodiments, the CXL switch 1905 may be connected to a PCIe device 1970 or an accelerator 1980 for supporting multiple functions, and the PCIe device 1970 or the accelerator 1980 may communicate with the first CPU 1910 , the second CPU 1920 , the GPU 1930 , and the NPU 1940 through the CXL switch 1905 , or may access a storage space STR including a CXL storage 1950 and a CXL memory 1960 .

[0159] In some example embodiments, CXL switch 1905 may be connected to an external network 1990 (or fabric), and may communicate with an external server through external network 1990 (or fabric).

[0160] In some example embodiments, the computing system 1900 may further include at least one of a CXL storage and a CXL memory, and at least one of the CXL storage and the CXL memory may be used for data scattering.

[0161] Fig. 20 A block diagram of a server of an application computing system according to an embodiment is shown.

[0162] refer to Fig. 20 , server 2000 collects a variety of data and provides services, and may be referred to as a data center or a data storage center. Server 2000 may be a system for operating a search engine and a database, and may be a computing system used by a company such as a bank or a government organization. Server 2000 may include application servers 2110 to 2130 and storage servers 2210 to 2230. The number of application servers 2110 to 2130 and the number of storage servers 2210 to 2230 may be selected in various ways according to the embodiment, and the number of application servers 2110 to 2130 (d, d is an integer greater than 1) and the number of storage servers 2210 to 2230 (e, e is an integer greater than 1) may be different from each other.

[0163] Now, the configuration of the first storage server 2210 may be mainly described. The application servers 2110 to 2130 and the storage servers 2210 to 2230 may have similar configurations, and the application servers 2110 to 2130 and the storage servers 2210 to 2230 may communicate with each other through a network (NT) 2205 .

[0164] The first storage server 2210 may include a processor 2211, a memory 2212, a switch 2213, a CXL memory 2214, a storage device 2215, and a network interface card (NIC) 2216. The processor 2211 may control the general operation of the first storage server 2210, and may access the memory 2212 to execute instructions loaded on the memory 2212 or process data. The memory 2212 may be a double data rate synchronous DRAM (DDR SDRAM), a hybrid memory cube (HMC), an HBM, a DIMM, an optane DIMM, and / or a non-volatile DIMM (NVMDIMM). The processor 2211 and the memory 2212 may be directly connected to each other, and the number of processors 2211 and the number of memories 2212 included in a storage server 2210 may be selected in a variety of ways.

[0165] In some example embodiments, the processor 2211 and the memory 2212 may provide a processor-memory pair. In some example embodiments, the number of processors 2211 and the number of memories 2212 may be different from each other. The processor 2211 may include a single-core processor or a multi-core processor. The description of the storage server 2210 may be similarly applied to each application server 2110 to 2130.

[0166] The switch 2213 may arbitrate or route communications between various components included in the first storage server 2210. In some example embodiments, the switch 2213 may be a CXL switch implemented based on a CXL interface.

[0167] The CXL memory 2214 may be connected to the switch 2213. In some example embodiments, the CXL memory 2214 may be used as a memory expander for the processor 2211. Alternatively, the CXL memory 2214 may be allocated as a dedicated memory or a buffer memory of the storage device 2215.

[0168] The storage device 2215 may include a CXL interface circuit (CXL_IF) 2217, a controller (CTRL) 2218, and a NAND flash memory (NAND) 2219. The storage device 2215 may store data or output the stored data according to a request of the processor 2211. In some example embodiments, the storage device 2215 may receive at least one predetermined or selectable desired area of ​​the CXL memory 2214 as an exclusive area, and may use the exclusive area as a buffer memory.

[0169] In some example embodiments, the storage device 2215 may be a reference Figures 2 to 12 In an embodiment, CXL memory 2214 may be a reference to CXL memory devices 130 and 140. Fig.13 and Fig.18 The CXL memory devices 230, 240, and 1830 described above are described. For example, when the usage ratio is greater than a reference value, the storage device 2215 (or the CXL memory 2214) may determine the CXL memory 2214 (or the storage device 2215) as a target device. The storage device 2215 (or the CXL memory 2214) may store data to the target device and may store the data. Therefore, a data bottleneck phenomenon according to a read request of a host including the processor 2211 may be alleviated, and data skew caused by the data bottleneck phenomenon may be alleviated.

[0170] A network interface card (NIC) 2216 may be connected to the switch 2213. The NIC 2216 may communicate with other storage servers 2220 to 2230 or other application servers 2110 to 2130 through the network 2205.

[0171] In some example embodiments, the NIC 2216 may include a network interface card and a network adapter. The NIC 2216 may be connected to the network 2205 through a wired interface, a wireless interface, a Bluetooth interface, and an optical interface. The NIC 2216 may include an internal memory, a digital signal processor (DSP), and a host bus interface, and may be connected to the processor 2211 and / or the switch 2213 through the host bus interface. In some example embodiments, the NIC 2216 may be combined into at least one of the processor 2211, the switch 2213, and the storage device 2215.

[0172] In some example embodiments, the network 2205 may be implemented with Fibre Channel (FC) or Ethernet. In this case, FC is a medium for high-speed data transmission, and an optical switch may be used to provide high performance / high availability. Depending on the access method of the network 2205, the storage servers 2210 to 2230 may be provided as file storage, block storage, or object storage.

[0173] In some example embodiments, the network 2205 may be a storage-only network, such as a storage area network (SAN). For example, the SAN may be an FC-SAN using an FC network and implemented according to the FC protocol (FCP). For another example, the SAN may be an IP-SAN using a TCP / IP network and implemented according to the SCSI or Internet SCSI (iSCSI) protocol over TCP / IP. In some example embodiments, the network 2205 may be a general-purpose network, such as a TCP / IP network. For example, the network 2205 may be implemented according to protocols such as Fibre Channel over Ethernet (FCoE), Network Attached Storage (NAS), or NVMe over Fiber (NVMe-oF).

[0174] In some example embodiments, at least one of the application servers 2110 to 2130 may store data requested to be stored in a user or client in one of the storage servers 2210 to 2230 through the network 2205. At least one of the application servers 2110 to 2130 may obtain data requested to be read by a user or client from one of the storage servers 2210 to 2230 through the network 2205. For example, at least one of the application servers 2110 to 2130 may be implemented with a network server or a database management system (DBMS).

[0175] In some example embodiments, at least one of the application servers 2110 to 2130 may access a memory, CXL memory, or storage device included in another application server through the network 2205, or may access a memory, CXL memory, or storage device included in the storage servers 2210 to 2230 through the network 2205. Therefore, at least one of the application servers 2110 to 2130 may perform various operations on data stored in other application servers and / or storage servers 2210 to 2230. For example, at least one of the application servers 2110 to 2130 may execute instructions for transferring or copying data between the application servers and / or storage servers 2210 to 2230. In this case, the data may pass through the memory or CXL memory of the storage servers 2210 to 2230 from the storage device of the storage servers 2210 to 2230, or may be directly moved to the memory or CXL memory of the application servers 2110 to 2130. The data moved through the network 2205 may be encrypted for security or privacy.

[0176] In some example embodiments, a storage device included in at least one of the application servers 2110 to 2130 and the storage servers 2210 to 2230 may receive a CXL memory included in at least one of the application servers 2110 to 2130 and the storage servers 2210 to 2230 as an exclusive area, and may use the exclusive area as a buffer memory. For example, a storage device 2215 included in the storage server 2210 may receive a CXL memory included in another storage server (e.g., 2230), and may access the CXL memory included in another storage server (e.g., 2230) through the switch 2213 and the NIC 2216. In this case, data of the storage device 2215 of the first storage server 2210 may be stored in the CXL memory of the storage server 2230. That is, the storage device and the CXL memory of the server 2000 according to the present disclosure may be connected and implemented in various ways.

[0177] In some example embodiments, reference Figures 1 to 20 Each constituent element described or a combination of two or more constituent elements may be implemented as a digital circuit, a programmable or non-programmable logic device or array, an application specific integrated circuit (ASIC), or the like.

[0178] While the inventive concepts have been described in conjunction with what are presently considered to be exemplary embodiments, it is to be understood that the inventive concepts are not limited to the disclosed exemplary embodiments, but, on the contrary, are intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A Compute Express Link (CXL) device for managing memory resources, comprising: A memory configured to store first data; as well as The control logic is configured to calculate a first usage ratio of the CXL device, and is configured to output second data as a part of the first data stored in the CXL device to the external shared memory through the CXL interface in response to the first usage ratio being greater than a reference value.

2. The CXL device according to claim 1, wherein The control logic obtains an address of the shared memory storing the second data, and writes the address into the mapping table.

3. The CXL device according to claim 2, wherein The control logic is configured to communicate with the host via the CXL.io protocol and to communicate with the shared memory via the CXL.mem protocol; The control logic is configured to, in response to receiving a read request for the first data from the host, send third data other than the second data among the first data to the host, and send a request signal for requesting the shared memory to send the second data to the host to the shared memory, and The request signal includes the address.

4. The CXL device of claim 3, wherein The control logic is configured to determine whether the shared memory has completed storing the second data, The control logic is configured to, in response to the control logic receiving a read request for the first data from the host and the shared memory having completed storing the second data, send a request signal to the shared memory for requesting the shared memory to send the second data to the host, and The control logic is configured to send the first data to the host in response to the shared memory not completing storing the second data.

5. The CXL device of claim 1, wherein The control logic is configured to obtain a first transmission rate between the CXL device and the host, and the control logic is configured to determine a ratio between the first data and the second data based on the first transmission rate.

6. The CXL device of claim 5, wherein The control logic is configured to obtain a second transfer rate between the host and the shared memory, and is configured to determine a ratio based on the first transfer rate and the second transfer rate.

7. The CXL device of claim 6, wherein The control logic is configured to determine a ratio of the first data to the second data such that "(first transmission rate) / (first transmission rate+second transmission rate)" corresponds to "size of the second data / size of the first data".

8. The CXL device of claim 1, wherein The control logic is configured to determine a read request number of data stored in the memory, and is configured to determine data having a maximum read request number as first data.

9. The CXL device of claim 8, wherein The control logic is configured to determine data having a second largest number of read requests as third data, and is configured to output a portion of the third data to the shared memory.

10. The CXL device of claim 1, wherein The control logic is configured to obtain a first time for sending first data to the host, obtain a second transmission rate between the host and the shared memory, and determine whether to output the second data based on the first time, the second transmission rate and the size of the second data.

11. The CXL device of claim 10, wherein The control logic is configured to calculate a second time of the shared memory based on a second transmission rate and a size of the second data, output the second data to the shared memory in response to the first time being longer than the second time, and not output the second data to the shared memory in response to the first time being equal to or shorter than the second time.

12. The CXL device of claim 1, wherein The control logic is configured to obtain a first time for sending first data to a host, obtain a first transmission rate between the CXL device and the host, obtain a second transmission rate between the host and a shared memory, and determine whether to output the second data based on the first time, the first data, the second data, the first transmission rate, and the second transmission rate.

13. The CXL device of claim 12, wherein The control logic is configured to calculate a second time for sending third data based on a first transmission rate and a size of the third data, the third data being other than the second data in the first data, calculate a third time for sending the second data based on a second transmission rate and a size of the second data, output the second data to a shared memory in response to the first time being longer than the sum of the second time and the third time, and not output the second data to the shared memory in response to the first time being equal to or shorter than the sum of the second time and the third time.

14. The CXL device of claim 1, wherein The control logic is configured to obtain a second usage ratio of the CXL devices shared through the CXL interface, and determine one of the CXL devices as a shared memory based on the second usage ratio.

15. An electronic device for managing memory resources, comprising: a first CXL device configured to store second data as part of the first data to a second CXL device in response to a read request for the first data being greater than a reference number; a second CXL device configured to store second data in response to a request from the first CXL device; as well as The switch is configured to connect the first CXL device and the second CXL device.

16. The electronic device according to claim 15, wherein The first CXL device is configured to send second data to the second CXL device in response to the number of read requests within a reference time being greater than a reference number.

17. The electronic device according to claim 15, wherein The first CXL device is configured to receive a read request from the host by using the CXL.io protocol, and is configured to send second data to the second CXL device by using the CXL.mem protocol.

18. The electronic device according to claim 17, wherein In response to receiving a read request for the first data from the host, the first CXL device is configured to send third data except the second data among the first data to the host by using the CXL.io protocol, and the first CXL device is configured to request transmission of the second data from the second CXL device by using the CXL.mem protocol, and The second CXL device is configured to send second data to the host by using the CXL.io protocol in response to the transmission request of the first CXL device.

19. The electronic device according to claim 18, wherein The first CXL device is configured to send a transmission completion signal of the first data to the host when the second CXL device completes the transmission of the second data.

20. A method for storing data, comprising: In response to the device receiving a first read request for first data from the host, sending the first data to the host; In response to the received number of first read requests being greater than a reference number, selecting one of the shared memories; determining second data as a portion of the first data based on a size of the first data, a first transmission rate of the selected shared memory, and a second transmission rate of the device; writing the second data into the selected shared memory; as well as In response to receiving a second read request for the first data from the host, transmission of the second data is requested to the selected shared memory, and third data other than the second data among the first data is sent to the host.

Images