Method for grouping embedded vectors in storage device, storage device and storage system

Abstract

The invention discloses a method for grouping embedded vectors in a storage device, the storage device and a storage system. In one embodiment, a method for grouping embedded vectors in a storage device includes receiving, at the storage device, an inferred query request including a first access index associated with a first embedded vector, determining a first classification of the first access index using a classification model, the method includes receiving, at a storage device, a first classification of a first access index associated with a second embedding vector, searching, in a cache of the storage device, a first embedding vector based on the first classification, receiving, at the storage device, a second access index associated with the second embedding vector, determining, using a classification model, a second classification of the second access index, and searching, in a persistent storage of the storage device, a second embedding vector based on the second classification.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 534,445, filed on August 24, 2023, entitled “REDUCING EMBEDDING TABLESREAD LATENCY IN DLRM INFERENCE VIA GROUPING / CLUSTERING METHODOLOGY AND LEVERAGING COMPUTE EXPRESS LINK (CXL) SSD,” and U.S. Application No. 18 / 497,250, filed on October 30, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] Aspects according to some embodiments of the present disclosure relate to a method for grouping embedding vectors in a storage device, a storage device, and a storage system. Background Art

[0003] In the field of computer storage, a system may include a host computer and one or more storage devices connected to (e.g., communicatively coupled to) the host computer. Such computer storage systems have become increasingly popular, in part to allow many different users to share the computing resources of the system. As the number of users of such systems and the number and complexity of applications running on such systems have increased, storage requirements have also increased over time.

[0004] Therefore, there may be a need for methods, systems, and devices suitable for improving the use of storage devices in storage systems.

[0005] This Background section is intended only to provide context, and the disclosure of any embodiments or concepts in this section does not constitute an admission that such embodiments or concepts are prior art. Summary of the invention

[0006] Aspects of some embodiments of the present disclosure relate to computer storage systems and provide improvements to machine learning (ML).

[0007] According to some embodiments of the present disclosure, a method is provided, the method comprising: receiving, at a storage device, an inference query request including a first access index associated with a first embedded vector; determining, using a classification model, a first classification of the first access index; searching, based on the first classification, a cache of the storage device for the first embedded vector; receiving, at the storage device, a second access index associated with a second embedded vector; determining, using the classification model, a second classification of the second access index; and searching, based on the second classification, a persistent storage device of the storage device for the second embedded vector.

[0008] The first classification may indicate a first access frequency, and the second classification may indicate a second access frequency that is less than the first access frequency.

[0009] Determining the first classification may include comparing an access frequency associated with the first access index with a reference access frequency; and determining the second classification may include comparing an access frequency associated with the second access index with the reference access frequency.

[0010] Determining the first classification may include generating an output from a machine learning (ML) grouping model based on the first access index.

[0011] Determining the second classification may include generating an output from a machine learning (ML) grouping model based on the second access index.

[0012] The storage device may be configured to communicate with the host via a cache coherence protocol.

[0013] The second embedded vector may be prevented from being stored in a cache of the storage device based on a reference access frequency.

[0014] According to some other embodiments of the present disclosure, a storage device is provided, which includes a cache memory and a persistent memory, wherein the storage device is configured to: receive an inference query request including a first access index associated with a first embedded vector; determine a first classification of the first access index using a classification model; search the cache memory for the first embedded vector based on the first classification; receive a second access index associated with a second embedded vector; determine a second classification of the second access index using the classification model; and search the persistent memory for the second embedded vector based on the second classification.

[0015] The first classification may indicate a first access frequency, and the second classification may indicate a second access frequency that is less than the first access frequency.

[0016] The process of determining the first classification may include comparing the access frequency associated with the first access index with a reference access frequency; and the process of determining the second classification may include comparing the access frequency associated with the second access index with the reference access frequency.

[0017] Determining the first classification may include generating an output from a machine learning (ML) grouping model based on the first access index.

[0018] Determining the second classification may include generating an output from a machine learning (ML) grouping model based on the second access index.

[0019] The storage device may be configured to communicate with the host via a cache coherence protocol.

[0020] The second embedded vector may be prevented from being stored in the cache memory based on the reference access frequency.

[0021] According to some other embodiments of the present disclosure, a system is provided, the system comprising a processor and a memory storing instructions, the instructions, when executed by the processor, causing the processor to perform: receiving an inference query request including a first access index associated with a first embedded vector; determining a first classification of the first access index using a classification model; searching for the first embedded vector in a cache of a storage device based on the first classification; receiving a second access index associated with a second embedded vector; determining a second classification of the second access index using the classification model; and searching for the second embedded vector in a persistent storage device of the storage device based on the second classification.

[0022] The first classification may indicate a first access frequency, and the second classification may indicate a second access frequency that is less than the first access frequency.

[0023] The process of determining the first classification may include comparing the access frequency associated with the first access index with a reference access frequency, and the process of determining the second classification may include comparing the access frequency associated with the second access index with the reference access frequency.

[0024] Determining the first classification may include generating an output from a machine learning (ML) grouping model based on the first access index.

[0025] Determining the second classification may include generating an output from a machine learning (ML) grouping model based on the second access index.

[0026] The second embedded vector may be prevented from being stored in a cache of the storage device based on a reference access frequency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0028] Figure 1 is a system diagram depicting a system for grouping embedding vectors in storage according to some embodiments of the present disclosure.

[0029] Figure 2 is a diagram depicting an inferred query request (IQR) and an embedding table according to some embodiments of the present disclosure.

[0030] Figure 3 is a diagram depicting a Pareto distribution in a time-local graph for access indexes according to some embodiments of the present disclosure.

[0031] Figure 4is a flow chart depicting a method of classifying access indexes using a reference frequency clustering model according to some embodiments of the present disclosure.

[0032] Figure 5 is a flow chart depicting a method of classifying access indexes using an ML clustering model according to some embodiments of the present disclosure.

[0033] Figure 6 is a flowchart depicting a method of grouping embedding vectors in a storage device according to some embodiments of the present disclosure.

[0034] Throughout the several views of the accompanying drawings, corresponding reference symbols indicate corresponding components. It will be appreciated by those skilled in the art that the elements in the accompanying drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some of the elements, layers, and regions in the accompanying drawings may be exaggerated relative to other elements, layers, and regions to help improve the clarity and understanding of the various embodiments. In addition, common but well-understood elements and components that are not relevant to the description of the embodiments may not be shown to facilitate less obstructed views of these various embodiments and to make the description clear. DETAILED DESCRIPTION

[0035] By referring to the detailed description of one or more embodiments and the accompanying drawings, the aspects of the present disclosure and the methods for realizing the same may be more easily understood. Hereinafter, the embodiments will be described in more detail with reference to the accompanying drawings. However, the described embodiments may be implemented in various different forms and should not be construed as being limited to the embodiments shown herein. On the contrary, these embodiments are provided as examples so that the present disclosure will be thorough and complete, and the aspects of the present disclosure will be fully conveyed to those skilled in the art. Therefore, the description of the processing, elements and techniques that are not necessary for a comprehensive understanding of the aspects and features of the present disclosure by those of ordinary skill in the art may be omitted.

[0036] Unless otherwise specified, the same reference numerals, symbols or combinations thereof represent the same elements throughout the drawings and written descriptions, and therefore, their descriptions will not be repeated. In addition, components not related to the description of the embodiments may not be shown to make the description clear. In the drawings, the relative sizes of elements, layers and regions may be exaggerated for clarity.

[0037] In the detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. However, it is apparent that the various embodiments can be practiced without these specific details or with one or more equivalent arrangements.

[0038] It will be understood that although the terms "zeroth", "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the spirit and scope of the present disclosure, the first element, first component, first region, first layer or first part described below may be referred to as the second element, second component, second region, second layer or second part.

[0039] It will be understood that when an element or component is referred to as being "on" another element or component, "connected to" or "coupled to" another element or component, it may be directly on, directly connected to or coupled to the other element or component, or one or more intermediate elements or components may be present. However, "direct connection / direct coupling" refers to a component being directly connected or coupled to another component without an intermediate component. At the same time, other expressions describing the relationship between components (such as "between...", "immediately between..." or "adjacent to..." and "close to...") may be similarly interpreted. In addition, it will also be understood that when an element or component is referred to as being "between" two elements or components, it may be the only element or component between the two elements or components, or one or more intermediate elements or components may also be present.

[0040] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to also include the plural form. It will also be understood that the terms "comprising", "having" and "including" when used in this specification indicate the presence of stated features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their groups. As used herein, each of the terms "or" and "and / or" includes any and all combinations of one or more of the associated listed items.

[0041] For the purposes of this disclosure, expressions such as "at least one of..." when following a list of elements modify the entire list of elements rather than modifying the individual elements in the list. For example, "at least one of X, Y, and Z" and "at least one selected from the group consisting of X, Y, and Z" may be interpreted as only X, only Y, only Z, or any combination of two or more of X, Y, and Z (such as, for example, XYZ, XYY, YZ, and ZZ).

[0042] As used herein, the terms "substantially," "about," "approximately," and similar terms are used as terms of approximation rather than terms of degree, and are intended to account for the inherent deviations in measured or calculated values ​​that one of ordinary skill in the art would recognize. As used herein, the terms "about" or "approximately" include the stated value and represent within an acceptable range of deviation for a particular value determined by one of ordinary skill in the art taking into account the measurement in question and the errors associated with the measurement of a particular quantity (i.e., the limitations of the measurement system). For example, "about" may mean within one or more standard deviations or within ±30%, ±20%, ±10%, ±5% of the stated value. In addition, the use of "may" when describing embodiments of the present disclosure refers to "one or more embodiments of the present disclosure."

[0043] When one or more embodiments can be implemented differently, a specific processing order can be performed differently from the described order. For example, two consecutively described processes can be performed substantially simultaneously or in the reverse order of the described order.

[0044] Any component or any combination of components described (e.g., in any system diagram included herein) may be used to perform one or more of the operations of any flowchart included herein. In addition, (i) the operations are merely examples and may involve various additional operations not explicitly included, and (ii) the temporal order of the operations may vary.

[0045] The electronic or electrical devices and / or any other related devices or components according to the embodiments of the present disclosure described herein may be implemented using any suitable hardware, firmware (e.g., an application specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, the various components of these devices may be formed on an integrated circuit (IC) chip or on separate IC chips. In addition, the various components of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on a substrate.

[0046] In addition, the various components of these devices can be processes or threads running on one or more processors in one or more computing devices, and one or more processors in one or more computing devices execute computer program instructions and interact with other system components for performing the various functions described herein. Computer program instructions are stored in a memory that can be implemented in a computing device using a standard memory device (such as, for example, a random access memory (RAM)). Computer program instructions may also be stored in other non-temporary computer-readable media (such as, for example, a CD-ROM, a flash drive, etc.). In addition, those skilled in the art should recognize that without departing from the spirit and scope of the embodiments of the present disclosure, the functions of various computing devices may be combined or integrated into a single computing device, or the functions of a particular computing device may be distributed on one or more other computing devices.

[0047] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which the inventive concept belongs. It will also be understood that terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or in this specification, and should not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.

[0048] As described above, in the field of computer storage, a system may include a host and one or more storage devices (e.g., a solid-state drive SSD) communicatively coupled to the host. The system may be used to process a machine learning workload (e.g., an inference workload). For example, a machine learning workload may be a deep learning recommendation model (DLRM) inference workload including many (e.g., billions) of embedding vectors (EVs). As used herein, an "embedding vector" refers to a data structure used in machine learning to provide a numerical representation of an object for processing by a neural network (e.g., a multilayer perceptron (MLP)). The embedding vector may be classified into different embedding tables (EMBs), and different embedding tables (EMBs) occupy a large amount of space in a memory (e.g., in a persistent memory). The embedding vector may be requested to be processed as part of the inference workload by means of an inference query request. As used herein, an "inference query request" refers to a request including an access index corresponding to a specific embedding vector. An inference query request may cause a batch of embedding vectors to be extracted from memory using an access index, each access index being used to locate a corresponding embedding vector. Persistent memory (also known as non-volatile memory, such as NAND flash memory, etc.) can be considered a relatively slow form of memory compared to volatile memory types (e.g., random access memory (RAM)) used for cache memory. Therefore, fetching the embedding vector only from persistent memory may result in increased latency, which may slow down the processing of inference workloads.

[0049] In order to reduce latency in a flexible manner, aspects of embodiments of the present disclosure provide improvements to such systems by grouping embedding vectors in fast memory (e.g., cache memory) and slow memory (e.g., persistent storage device) based on using a clustering model (also referred to as a grouping model or a classification model) and utilizing a Pareto distribution for access indexes to exclude less frequently accessed embedding vectors from fast memory. In some embodiments, the clustering model can be a reference frequency grouping model. In some embodiments, the clustering model can be a machine learning (ML) grouping model.

[0050] Figure 1 is a system diagram depicting a system for grouping embedding vectors in storage according to some embodiments of the present disclosure.

[0051] Reference Figure 1, the system 1 may include a host 100 that is communicatively coupled to a storage device 200. In some embodiments, the host 100 may be communicatively coupled to the storage device 200 via a cache coherence protocol such as Compute Express Link (CXL) or the like. For example, the storage device 200 may be a CXL solid state drive (SSD). The storage device 200 may include a cache 230 (e.g., a cache memory) and a persistent storage device 240. As discussed above, the memory type of the cache 230 may be a memory type (e.g., a dynamic random access memory (DRAM) cache) that is faster than the memory type of the persistent storage device (e.g., NAND flash memory). The storage device 200 may include a cache controller 220 for managing data storage within the cache 230 and / or the persistent storage device 240. The persistent storage device 240 may be used to store an embedded vector table EMB. Each embedded vector table EMB may include an embedded vector EV. The embedded vector may correspond to a sparse feature for processing by an application 111 running on the host 100. The application 111 may be a DLRM for processing a workload WL (e.g., an inference workload) on the host 100. The application 111 may include a neural network for processing dense features DF and sparse features to generate inferences. The neural network may be a multi-layer perceptron (MLP). The MLP may include a bottom MLP 151 and a top MLP 155.

[0052] To generate an inference based on a specific embedding vector EV, the application 111 may send an inference query request IQR to the storage device 200. The inference query request IQR may include a sub-request 10 (see Figure 2 ), the sub-request 10 is associated with an embedding vector EV and comprises an access index 20 for locating the corresponding embedding vector EV (see Figure 2 ). The storage device 200 may receive the inference query request IQR at the cache controller 220. The cache controller 220 may include a clustering model 222. The clustering model 222 may determine whether the storage device 200 may search for the embedding vector EV in the cache 230 or the persistent storage device 240. The clustering model 222 may determine the classification of the embedding vector EV associated with the access index 20 based on whether the access index is a frequently accessed index. If the clustering model 222 determines that the access index 20 is frequently accessed, the storage device 200 may search for the embedding vector EV in the cache 230. If the embedding vector EV is in the cache 230, the storage device 200 may send the embedding vector EV to the embedding accumulator 250 at the host 100. If the embedding vector EV is not in the cache 230, the storage device may search for and obtain the embedding vector EV from the persistent storage device 240 to send to the embedding accumulator 250.

[0053] On the other hand, if the clustering model 222 determines that the access index 20 is not frequently accessed, the storage device 200 may search (e.g., directly search) and obtain the embedding vector EV from the persistent storage device 240 to send to the embedding accumulator 250. In some embodiments, the storage device 200 may prevent (e.g., exclude) the embedding vector EV that is not frequently accessed from being stored in the cache 230. The MLP may process the embedding vector EV with the dense feature DF received by the embedding accumulator 250 by means of interaction 153 (e.g., product, summation, etc.) to generate an inference.

[0054] In some embodiments, clustering model 222 may be as follows: Figure 4 In some embodiments, the clustering model 222 may be as follows: Figure 5 ML grouping model for discussion.

[0055] Figure 2 is a diagram depicting an inferred query request (IQR) and an embedding table according to some embodiments of the present disclosure.

[0056] Reference Figure 2 , the inference query request IQR may include many sub-requests 10. Each sub-request 10 may include an access index 20 for locating a corresponding embedding vector EV in a corresponding embedding table EMB. The clustering model 222 may process the sub-requests 10 one by one. Therefore, the first received access index 20 and the second received access index 20 may come from the same inference query request IQR or from different inference query requests IQR. Each inference query request IQR may have a given batch size BS. Each inference query request IQR may include multiple small batches MB with a given small batch size MBS. Each inference query request IQR may be described according to the number of indexes NIL searched each time. Each embedding table EMB may have a given embedding table size EMBS. Each embedding table EMB may be described according to a sparse feature size SFS.

[0057] Figure 3 is a diagram depicting a Pareto distribution in a temporal locality graph for access indexes according to some embodiments of the present disclosure.

[0058] Reference Figure 3 ,DLRM infers that workloads may exhibit a Pareto distribution, where a relatively small percentage of access indexes are requested by the workload most of the time. For example, Figure 3The graph depicted in shows a non-limiting example of "where 20% of all access indexes (see the vertical dashed line in contact with the x-axis of the graph) account for more than 80% of all index access requests (see the horizontal dashed line in contact with the y-axis of the graph)". In some embodiments, the Pareto distribution associated with the access indexes of the storage device may serve as a basis for "determining whether a given access index is classified as falling into the high frequency access group HFA or the low frequency access group LFA". In some embodiments, the access indexes classified as falling into the high frequency access group HFA may be stored in the cache 230. In some embodiments, the access indexes classified as falling into the low frequency access group LFA may be excluded from "storage in the cache 230" and may be stored only in the persistent storage device 240. That is, the storage device 200 (see Figure 1 ) can prevent access indexes classified as falling within the low frequency access group LFA from being stored in the cache 230. Therefore, when compared to storing some indexes falling within the low frequency access group LFA in the cache 230 or storing some indexes falling within the high frequency access group HFA only in the persistent storage device 240, the latency associated with the workload can be improved.

[0059] The inflection point of the Pareto distribution may correspond to the medium frequency access group MFA, which may include access indexes that may be classified as the high frequency access group HFA or the low frequency access group LFA according to specific aspects of the application or system. For example, if the size of the cache 230 is more limited in a given system, a reference access frequency (e.g., a threshold access frequency) between the high frequency access group HFA and the low frequency access group LFA may be selected to exclude more access indexes falling within the medium frequency access group MFA from being “stored in the cache 230”.

[0060] Figure 4 is a flow chart depicting a method of classifying access indexes using a reference frequency clustering model according to some embodiments of the present disclosure.

[0061] The clustering model 222a can classify the access indexes by forming an understanding of the Pareto distribution based on one or more initial access indexes (e.g., one or more sample access indexes) and determining (e.g., manually and empirically) a reference access frequency (e.g., a threshold access frequency) based on the Pareto distribution. The threshold access frequency can be compared with any given access index to determine whether the access index falls into a high frequency access group HFA or a low frequency access group LFA (see Figure 3 ).

[0062] The method 4000 for classifying access indexes using the reference frequency clustering model 222a may include the following example operations. Based on receiving the first access index 20 associated with the first embedding vector EV, the storage device 200 (see Figure 1 and Figure 2 ) may determine a reference access frequency (operation 4001). The storage device 200 may receive a second access index 20 associated with a second embedding vector EV, and compare the access frequency of the second access index 20 with the reference access frequency (operation 4002). The storage device 200 (e.g., the clustering model 222) may determine whether the second access index 20 is eligible to be classified under the first classification based on the comparison of operation 4002 (operation 4003). If the storage device 200 determines that the second access index 20 is eligible to be classified under the first classification, the storage device 200 may search for the second embedding vector EV in the cache 230 (operation 4004a). If the storage device 200 determines that the second access index 20 is not eligible to be classified under the first classification (e.g., eligible to be classified under a second classification different from the first classification), the storage device 200 may search for the second embedding vector EV in the persistent storage device 240 (operation 4004b). It should be understood that the method 4000 may be performed using any suitable frequency-based clustering model known to those of ordinary skill in the art.

[0063] Figure 5 is a flow chart depicting a method of classifying access indexes using an ML clustering model according to some embodiments of the present disclosure.

[0064] Reference Figure 5 , the method 5000 for classifying access indexes using the ML clustering model 222b may include the following example operations. An initial ML clustering model 222b may be constructed and trained to infer (e.g., estimate or predict) whether a cache 230 or a persistent storage device 240 (see Figure 1) finds (e.g., locates) an embedding vector corresponding to the received access index in the cache 230 (operation 5001). The storage device 200 may receive a first access index associated with a first embedding vector EV (operation 5002). For example, the storage device 200 may receive the first access index after a cold start build period associated with the ML clustering model 222b (operation 5002). The storage device 200 may determine whether to search for the first embedding vector in the cache 230 based on the output generated by the ML clustering model 222b (operation 5003). If the output indicates that the first embedding vector EV can be found in the cache 230, the storage device 200 may search for the first embedding vector EV in the cache 230 (operation 5004a). If the output indicates that the first embedding vector EV can be found in the persistent storage device 240 (e.g., can be found only in the persistent storage device 240), the storage device 200 may search (e.g., can directly search) the first embedding vector EV in the persistent storage device 240 without checking the cache 230 (e.g., bypassing the cache 230) (operation 5004b). It should be understood that method 5000 may be performed using any suitable ML-based clustering model known to one of ordinary skill in the art.

[0065] Figure 6 is a flowchart depicting a method of grouping embedding vectors in a storage device according to some embodiments of the present disclosure.

[0066] Reference Figure 6 , the method 6000 may include the following example operations. In one embodiment, the storage device 200 (e.g., the cache controller 220 in the storage device 200) may be configured to perform the following example operations. The storage device 200 may receive an inference query request IQR including a first access index 20 associated with a first embedding vector EV (operation 6001). The storage device 200 may determine a first classification (e.g., high frequency access) of the first access index 20 using a clustering model 222 (e.g., a classification model) (operation 6002). Based on the first classification, the storage device 200 may search for the first embedding vector in the cache 230 (operation 6003). The storage device 200 may receive a second access index 20 associated with a second embedding vector EV (operation 6004). The storage device 200 may determine a second classification (e.g., low frequency access) of the second access index using the clustering model 222 (operation 6005). Based on the second classification, the storage device 200 may search for the second embedding vector EV in the persistent storage device 240 (operation 6006).

[0067] Therefore, aspects of some embodiments of the present disclosure may provide improvements in data storage by providing systems and methods for grouping embedded vectors in fast memory and slow memory by using a reference frequency clustering model or an ML grouping model, which may be flexibly applied to various storage devices to achieve improved cache hit rates, reduced storage device persistent memory lookups, and reduced overall embedded vector lookup latency.

[0068] The disclosed example embodiments may extend to, but are not limited to, the following statements: Statement 1. An example method comprises: receiving at a storage device an inference query request including a first access index associated with a first embedded vector, determining a first classification of the first access index using a classification model, searching a cache of the storage device for the first embedded vector based on the first classification, receiving at the storage device a second access index associated with a second embedded vector, determining a second classification of the second access index using the classification model, and searching a persistent storage device of the storage device for the second embedded vector based on the second classification.

[0069] Statement 2. An example method includes the method of statement 1, wherein the first classification indicates a first access frequency and the second classification indicates a second access frequency less than the first access frequency.

[0070] Statement 3. An example method includes a method according to any one of statements 1 and 2, wherein the step of determining a first classification includes: comparing an access frequency associated with a first access index with a reference access frequency, and the step of determining a second classification includes: comparing an access frequency associated with a second access index with a reference access frequency.

[0071] Statement 4. An example method comprises the method of any of statements 1 and 2, wherein the step of determining the first classification comprises: generating an output from a machine learning (ML) grouping model based on the first access index.

[0072] Statement 5. An example method comprises the method of any of statements 1, 2, and 4, wherein the step of determining the second classification comprises: generating an output from a machine learning (ML) grouping model based on the second access index.

[0073] Statement 6. An example method comprises the method of any of statements 1 to 5, wherein the storage device is configured to communicate with the host via a cache coherence protocol.

[0074] Statement 7. An example method comprises the method of any of statements 1 to 6, wherein the second embedded vector is prevented from being stored in a cache of the storage device based on a reference access frequency.

[0075] Statement 8. An example apparatus for performing the method of any one of statements 1 to 7 comprises: a storage device including a cache memory and a persistent memory, wherein the storage device is configured to perform the method of any one of statements 1 to 7.

[0076] Statement 9. An example system for performing the method of any one of statements 1 to 7 comprises: a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform the method of any one of statements 1 to 7.

[0077] While embodiments of the present disclosure have been particularly shown and described with reference to the embodiments described herein, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as set forth in the appended claims and their equivalents.

Claims

1. A method for grouping embedding vectors in a storage device, comprising: receiving, at a storage device, an inference query request including a first access index associated with a first embedding vector; determining a first classification of the first access index using the classification model; Based on the first classification, searching a cache of a storage device for a first embedding vector; receiving, at a storage device, a second access index associated with a second embedded vector; determining a second classification of the second access index using the classification model; as well as Based on the second classification, a second embedding vector is searched in a persistent storage device of the storage device.

2. The method according to claim 1, wherein: The first classification indicates a first access frequency; and The second classification indicates a second access frequency that is less than the first access frequency.

3. The method according to claim 1, wherein: The step of determining the first classification includes: comparing the access frequency associated with the first access index with a reference access frequency; and The step of determining the second classification comprises comparing the access frequency associated with the second access index with a reference access frequency.

4. The method according to claim 1, wherein: The step of determining a first classification includes generating an output from a machine learning grouping model based on the first access index.

5. The method according to claim 4, wherein: The step of determining a second classification includes generating an output from a machine learning grouping model based on the second access index.

6. The method according to any one of claims 1 to 5, wherein: The storage device is configured to communicate with the host via a cache coherence protocol.

7. The method according to any one of claims 1 to 5, wherein: The second embedded vector is prevented from being stored in a cache of the storage device based on a reference access frequency.

8. A storage device, comprising: Cache memory; as well as Persistent storage, Wherein, the cache controller in the storage device is configured as follows: receiving an inference query request including a first access index associated with a first embedding vector; determining a first classification of the first access index using the classification model; Based on the first classification, searching the cache memory for a first embedding vector; receiving a second access index associated with a second embedding vector; determining a second classification for the second access index using the classification model; and Based on the second classification, a second embedding vector is searched in the persistent memory.

9. The storage device according to claim 8, wherein: The first classification indicates a first access frequency; and The second classification indicates a second access frequency that is less than the first access frequency.

10. The storage device according to claim 8, wherein: The process of determining the first classification includes: comparing the access frequency associated with the first access index to a reference access frequency; and The process of determining the second classification includes comparing the access frequency associated with the second access index with a reference access frequency.

11. The storage device according to claim 8, wherein: The process of determining a first classification includes generating an output from a machine learning grouping model based on the first access index.

12. The storage device according to claim 11, wherein: The process of determining a second classification includes generating an output from a machine learning grouping model based on the second access index.

13. The storage device according to any one of claims 8 to 12, wherein: The storage device is configured to communicate with a host via a cache coherence protocol.

14. The storage device according to any one of claims 8 to 12, wherein: The second embedded vector is prevented from being stored in the cache memory based on the reference access frequency.

15. A storage system comprising: processor; as well as A memory storing instructions that, when executed by the processor, cause the processor to perform: receiving an inference query request including a first access index associated with a first embedding vector; determining a first classification of the first access index using the classification model; Based on the first classification, searching a cache of a storage device for a first embedding vector; receiving a second access index associated with a second embedding vector; determining a second classification for the second access index using the classification model; and Based on the second classification, a second embedding vector is searched in a persistent storage device of the storage device.

16. The storage system according to claim 15, wherein: The first classification indicates a first access frequency; and The second classification indicates a second access frequency that is less than the first access frequency.

17. The storage system according to claim 15, wherein: The process of determining the first classification includes: comparing the access frequency associated with the first access index to a reference access frequency; and The process of determining the second classification includes comparing the access frequency associated with the second access index with a reference access frequency.

18. The storage system according to claim 15, wherein: The process of determining a first classification includes generating an output from a machine learning grouping model based on the first access index.

19. The storage system according to claim 18, wherein: The process for determining the second classification includes: An output is generated from the machine learning grouping model based on the second access index.

20. The storage system according to any one of claims 15 to 19, wherein: The second embedded vector is prevented from being stored in a cache of the storage device based on a reference access frequency.

Images