Inference system and inference method

Abstract

The invention discloses an inference system and an inference method. The inference system comprises a first memory device for storing a plurality of historical query vectors; a second memory device; the first processor is used for converting a query into a query vector, searching a first type of vector from the plurality of historical query vectors stored in the first memory equipment, and pre-fetching a cluster to which the first type of vector belongs to the second memory equipment; and a second processor searching the first type of cluster center among the cluster centers of the clusters to which the first type of vector belongs, determining a vector most similar to the query vector based on the cluster among the clusters prefetched by the first processor to the second memory device, and determining a reasoning result based on the vector most similar to the query vector.

Description

Technical Field

[0001] This application relates to data processing technology, and more specifically, to reasoning systems and reasoning methods. Background Technology

[0002] Large Language Models (LLMs) are increasingly being applied across various fields. Retrieval Enhancement Generation (RAG) is a widely used algorithm for enhancing LLMs, where RAG improves inference quality by providing relevant context to the LLM's inference model. Typically, during the inference process of an LLM, large amounts of data transfer can lead to reduced inference efficiency. Summary of the Invention

[0003] According to the disclosed aspects, an inference system includes: a first memory device configured to store a plurality of historical query vectors corresponding to a plurality of historical queries; a second memory device; a first processor configured to: convert a query into a query vector, search for a first type vector among the plurality of historical query vectors stored in the first memory device, and prefetch the cluster to which the first type vector belongs from a plurality of clusters of vectors in a vector database belongs to the second memory device; and a second processor configured to: search for a first type cluster center among the cluster centers of the clusters to which the first type vector belongs found by the first processor, determine the vector most similar to the query vector based on the cluster to which the first type cluster center belongs from the clusters prefetched by the first processor to the second memory device, and determine an inference result based on the vector most similar to the query vector, wherein the similarity between the first type vector and the query vector is greater than a first threshold, and wherein the similarity between the first type cluster center and the query vector is greater than a cluster center threshold.

[0004] According to the disclosed aspects, an inference method includes: storing multiple historical query vectors corresponding to multiple historical queries via a first memory device; converting queries into query vectors via a first processor; searching for a first type vector among the multiple historical query vectors stored in the first memory device via the first processor; prefetching the cluster to which the first type vector belongs from multiple clusters of vectors in a vector database to a second memory device via the first processor; searching for a first type cluster center among the cluster centers of the clusters to which the first type vector belongs found by the first processor via a second processor; determining, via the second processor, the vector most similar to the query vector based on the cluster to which the first type cluster center belongs from the clusters prefetched by the first processor to the second memory device; and determining an inference result based on the vector most similar to the query vector via the second processor, wherein the similarity between the first type vector and the query vector is greater than a first threshold, and wherein the similarity between the first type cluster center and the query vector is greater than a cluster center threshold.

[0005] According to the disclosed aspect, a non-transitory computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor performs the above-described reasoning method.

[0006] In this embodiment, by reusing the search results of historical queries that are extremely similar to the new query, the transmission of cluster centers used for simple search and clusters used for cluster search, as well as the computation of simple search and cluster search, can be saved, thereby improving the efficiency of inference.

[0007] In this embodiment, by storing frequently used historical query vectors and their respective cluster centers in a second memory device, the number of cluster centers that need to be transferred for simple searches can be reduced.

[0008] In this embodiment, by storing frequently used vectors in a first memory device and infrequently used vectors in a third memory device, the transmission time of frequently used vectors can be reduced, thereby further improving the efficiency of inference.

[0009] In this embodiment, when the second processor fails to find a cluster center similar to the query vector, the first processor can perform a cluster search, thereby reducing the waiting time required to transmit clusters and further improving the efficiency of inference.

[0010] In this embodiment, when performing the calculation of the Nth layer of the large language model, the data required for the N+1th layer is transmitted to reduce transmission waiting time and further improve the efficiency of inference.

[0011] In this application, by storing as much intermediate data as possible for each layer, the operations of migrating intermediate data to and reading intermediate data from external devices can be reduced, thereby reducing input / output (IO) frequency and IO overhead.

[0012] In this application, compared to existing techniques that use cluster centers of all clusters to perform simple searches, using cluster centers of clusters belonging to historical queries that are relatively similar to the new query to perform simple searches reduces the number of cluster centers used for simple searches and reduces data transfer, thereby improving inference efficiency. In one or more examples, in this application, by prefetching clusters used for cluster searches to a second memory device, the second processor wait time is reduced, thereby further improving inference efficiency.

[0013] In this embodiment, by reusing the search results of historical queries that are extremely similar to the new query, the transmission of cluster centers used for simple search and clusters used for cluster search, as well as the computation of simple search and cluster search, can be saved, thereby further improving the efficiency of inference.

[0014] In this embodiment, by storing frequently used historical query vectors and their respective cluster centers in a second memory device, the number of cluster centers that need to be transferred for simple searches can be reduced.

[0015] In this embodiment, by storing frequently used vectors in a first memory device and infrequently used vectors in a third memory device, the transmission time of frequently used vectors can be reduced, thereby further improving the efficiency of inference.

[0016] In this embodiment, when the second processor fails to find a cluster center similar to the query vector, the first processor can perform a cluster search, thereby reducing the waiting time required to transmit clusters and further improving the efficiency of inference.

[0017] In this embodiment, when performing the calculation of the Nth layer of the large language model, the data required for the N+1th layer is transmitted to reduce transmission waiting time and further improve the efficiency of inference.

[0018] In this embodiment, by storing as much intermediate data as possible from the layers, the operations of migrating intermediate data to external devices and reading intermediate data from external devices can be reduced, thereby reducing IO frequency and IO overhead.

[0019] Further aspects and / or advantages of the inventive concept will be set forth in part in the description which follows, and in part will be obvious from the description and / or may be learned by practice of various exemplary embodiments. Attached Figure Description

[0020] The above and other objects, features and advantages of this disclosure will become clearer from the following detailed description taken in conjunction with the accompanying drawings.

[0021] Figure 1 and Figure 2 The RAG process based on relevant technologies is shown.

[0022] Figure 3 This is a block diagram illustrating an inference system according to some example embodiments.

[0023] Figure 4 This is a flowchart illustrating a reasoning method according to some example embodiments.

[0024] Figure 5 This is a flowchart illustrating a method of operating a first processor according to some example embodiments.

[0025] Figure 6 This is a flowchart illustrating a method for operating a second processor in a first case according to some example embodiments.

[0026] Figure 7This is a flowchart illustrating a method for operating a second processor in a second case according to some example embodiments.

[0027] Figure 8 This is a diagram illustrating a first memory device according to some example embodiments.

[0028] Figure 9 This is a diagram illustrating a second memory device according to some example embodiments.

[0029] Figure 10 This is a diagram illustrating a third memory device according to some example embodiments.

[0030] Figure 11 This is a schematic diagram illustrating how inference operations are performed in a first case according to some example embodiments.

[0031] Figure 12 This is a schematic diagram illustrating how inference operations are performed in a second case according to some example embodiments.

[0032] Figure 13 This is a flowchart illustrating the reasoning method according to some example embodiments in more detail.

[0033] Figure 14 This is a block diagram of an example computer system according to embodiments of the present disclosure. Detailed Implementation

[0034] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, upon understanding the disclosure of this application, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent. For example, the order of operations described herein is merely illustrative, except for operations that must be performed in a specific order, and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding the disclosure of this application. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0035] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein that will be clear upon understanding the disclosure of this application.

[0036] The structural or functional descriptions of the examples disclosed herein are intended for illustrative purposes only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to include various modifications, equivalents, and substitutions within the scope of the claims.

[0037] Although the terms “first” or “second” are used to describe various components, the components are not limited to the terms. These terms should only be used to distinguish one component from another. For example, within the scope of the claims based on the concept of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0038] It will be understood that when a component is referred to as being "connected to" another component, the component may be directly connected to or combined with the other component, or there may be an intermediate component.

[0039] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0040] Unless otherwise defined, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the examples pertain. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formalized sense.

[0041] The examples will be described in detail below with reference to the accompanying drawings. Regarding the reference numerals assigned to the elements in the drawings, it should be noted that the same elements will be represented by the same reference numerals, and redundant descriptions will be omitted.

[0042] Figure 1 The RAG process based on relevant technologies is shown.

[0043] Figure 1 This illustrates the execution of simple searches and cluster searches by a graphics processing unit (GPU). For example, a simple search can be a search for the cluster center most similar to the query vector, while a cluster search can be a search for the vector within a cluster most similar to the query vector.

[0044] First, when a query is received (for example, from a user), the central processing unit (CPU) can convert the query into a query vector and send the query vector to the GPU.

[0045] like Figure 1As shown, in operation S1, the CPU can send the cluster centers (or cluster center indices) of multiple clusters of vectors in the vector database to the GPU. For example, the vectors in the vector database can be divided into multiple clusters by clustering, and the cluster center of each cluster can be obtained.

[0046] In operation S2, the GPU can perform a simple search based on the cluster centers of each cluster. For example, the GPU can calculate the similarity between the cluster centers of each cluster and the query vector to determine at least one cluster center among multiple cluster centers that is most similar to the query vector. In one or more examples, the GPU can send at least one cluster center or the cluster number of at least one cluster center to the CPU.

[0047] In operation S3, the CPU can send at least one cluster (the index of all vectors of each in at least one cluster) to the GPU.

[0048] In operation S4, the GPU may perform a cluster search based on at least one cluster. For example, the GPU may compute the similarity between all vectors in each of at least one cluster and the query vector to determine at least one vector that is most similar to the query vector. In one or more examples, the GPU may send at least one vector to the CPU.

[0049] In operation S5, the CPU can send at least one of the context of at least one vector and the key-value cache (KVcache) to the GPU.

[0050] In operation S6, the GPU can determine the inference outcome based on the context of at least one vector and at least one execution large language model (LLM) in the KVcache.

[0051] Figure 2 The RAG process based on relevant technologies is shown.

[0052] Figure 2 This shows the CPU performing simple search and cluster search.

[0053] First, upon receiving a query, the CPU can convert the query into a query vector.

[0054] like Figure 2 As shown, in operation S1', the CPU can perform a simple search based on the cluster center of each of the multiple clusters of vectors in the vector database. For example, the CPU can calculate the similarity between the cluster center of each cluster and the query vector to determine at least one cluster center among the multiple cluster centers that is most similar to the query vector.

[0055] In operation S2', the CPU may obtain at least one cluster to which at least one cluster center belongs from a solid-state drive (SSD), and perform a cluster search based on at least one cluster. For example, the CPU may calculate the similarity between all vectors in each of the at least one cluster and the query vector to determine at least one vector that is most similar to the query vector.

[0056] In operation S3', the CPU can obtain the context of at least one vector and at least one KVcache from the SSD, and send the context of at least one vector and at least one KVcache to the GPU.

[0057] In operation S4', the GPU can perform LLM based on the context of at least one vector and at least one in the KVcache to determine the inference result.

[0058] As referenced above Figure 1 and Figure 2 As described, whether the search is performed by the GPU or the CPU, there is a significant amount of data transfer between the GPU and the CPU. The GPU must wait for this data transfer to complete before it can perform its own search and computation. This can lead to wasted GPU resources and low inference efficiency. The embodiments of this disclosure provide significant improvements over these drawbacks.

[0059] Figure 3 This is a block diagram illustrating an inference system according to some example embodiments.

[0060] like Figure 3 As shown, the inference system 1 may include a first processor 11, a second processor 12, a first memory device 21, and a second memory device 22. However, the embodiments are not limited thereto, and the inference system 1 may also include other devices.

[0061] According to one or more example embodiments, the first memory device 21 may store multiple historical query vectors (or indices of multiple historical query vectors) corresponding to multiple historical queries. For example, the multiple historical queries may be queries entered within a predetermined time period in the past from the current time (e.g., by a user, etc.). In one or more examples, the multiple historical queries may come from one or more users using multiple devices. For example, the index of each historical query vector includes the corresponding historical query vector, the cluster number of the cluster to which the corresponding historical query vector belongs, and the cluster center of the cluster to which the corresponding historical query vector belongs.

[0062] According to one or more example embodiments, the first processor 11 can receive a query and convert it into a query vector, search for a first type vector among a plurality of historical query vectors stored in the first memory device 21, and prefetch the cluster to which the first type vector belongs from a plurality of clusters of vectors in a vector database belongs to the second memory device 22. For example, the similarity between the first type vector and the query vector may be greater than a first threshold. For example, the similarity between the first type vector and the query vector may be less than a second threshold, and the second threshold is greater than the first threshold. In one or more examples, cosine similarity, Euclidean distance, or any other suitable similarity calculation known to those skilled in the art can be used to determine the similarity between two vectors.

[0063] In one or more examples, the first processor 11 may prefetch a third type vector from the cluster to which the first type vector belongs from the first memory device 21 to the second memory device 22, and prefetch a fourth type vector from the cluster to which the first type vector belongs from the third memory device to the second memory device 22. For example, the third type vector is a vector whose usage frequency is greater than a first usage frequency threshold, and the fourth type vector is a vector whose usage frequency is less than the first usage frequency threshold. In one or more examples, the first processor 11 may also prefetch a fourth type vector from the cluster to which the first type vector belongs from the third memory device to the first memory device 21.

[0064] According to one or more example embodiments, the second processor 12 can search for a first-type cluster center among the cluster centers of the cluster to which the first-type vector found by the first processor 11 belongs. For example, the similarity between the first-type cluster center and the query vector may be greater than a cluster center threshold. For example, the second processor 12 can perform a simple search based on the cluster centers of the cluster to which the first-type vector belongs. The second processor 12 can calculate the similarity between each cluster center of the cluster to which the first-type vector belongs and the query vector, and determine the cluster centers with a similarity greater than the cluster center threshold as the first-type cluster centers.

[0065] According to one or more example embodiments, the second processor 12 can look up the center of the first type cluster while the first processor 11 prefetches the cluster to which the first type vector belongs to the second memory device. In this regard, at least a portion of the lookup step can be executed in parallel with a portion of the cluster prefetching step. In one or more examples, the prefetching step and the lookup step may not start simultaneously. For example, the prefetching step may start before the lookup step, or the lookup step may start before the prefetching step.

[0066] According to one or more example embodiments, the second processor 12 may determine the vector most similar to the query vector based on the cluster to which the first type cluster center belongs among the clusters prefetched by the first processor 11 to the second memory device 22. For example, the second processor 12 may perform a cluster search based on the cluster to which the first type cluster center belongs. For example, the second processor 12 may search for the vector most similar to the query vector within the cluster to which the first type cluster center belongs.

[0067] According to one or more example embodiments, the second processor 12 may determine the inference result based on the vector most similar to the query vector.

[0068] In one or more examples, the first type vector found by the first processor 11 may be similar to the query vector, and therefore, the cluster center of the cluster to which the first type vector belongs will also be similar to the query vector. However, due to the computational discrepancy between the first processor 11 and the second processor 12, there is a small probability that the cluster center of the cluster to which the first type vector belongs will have a low similarity to the query vector (e.g., no first type cluster center is found). To avoid performing a simple search based on all cluster centers and then performing a cluster search when no first type cluster center is found (which may result in a large amount of data transfer and computation), in this application, when no first type cluster center is found, the first processor 11 may perform a cluster search based on the cluster to which the first type vector belongs.

[0069] In the examples described above, the first processor 11 may search for a first-type vector. To further reduce the amount of data transfer and computation caused by searching for the most similar vector to the query vector, the first processor 11 may search for a second-type vector (the second-type vector is more similar to the query vector than the first-type vector) among multiple historical query vectors. In response to finding the second-type vector, the first processor 11 may send at least one of the context of the second-type vector pre-stored in the first memory device 21 and the KVcache to the second processor 12, so that the second processor 12 directly determines the inference result based on the context of the second-type vector and at least one of the KVcache. In one or more examples, the first processor 11 may prefetch the cluster to which the first-type vector belongs to the second memory device 22 in response to finding the first-type vector among multiple historical query vectors and not finding the second-type vector.

[0070] In one or more examples, when neither a first-type vector nor a second-type vector exists among multiple historical query vectors, the above reference can be used. Figure 1 or Figure 2 The way the description is presented determines the result of the reasoning.

[0071] In one or more examples, the first processor 11 may include a central processing unit (CPU) and the second processor 12 may include a graphics processing unit (GPU). However, the examples are not limited to this, and the first processor 11 and the second processor 12 may include other processors.

[0072] In this embodiment, compared to existing techniques that use cluster centers of all clusters to perform a simple search, using cluster centers of clusters belonging to historical queries that are relatively similar to the new query to perform a simple search reduces the number of cluster centers used for the simple search and reduces data transfer, thereby improving inference efficiency. In one or more examples, in this embodiment, by prefetching clusters used for cluster search to a second memory device, the second processor wait time is reduced, thereby further improving inference efficiency.

[0073] In this embodiment, by reusing the search results of historical queries that are extremely similar to the new query, the transmission of cluster centers used for simple search and clusters used for cluster search, as well as the computation of simple search and cluster search, can be saved, thereby further improving the efficiency of inference.

[0074] In this embodiment, when the second processor fails to find a cluster center similar to the query vector, the first processor can perform a cluster search, thereby reducing the waiting time required to transmit clusters and further improving the efficiency of inference.

[0075] Figure 4 This is a flowchart illustrating a reasoning method according to some example embodiments.

[0076] According to one or more example embodiments, such as Figure 4 As shown, in operation S21, the first processor 11 can receive a query and convert the query into a query vector.

[0077] In operation S22, the first processor 11 can search for a first type vector among multiple historical query vectors stored in the first memory device 21. For example, the similarity between the first type vector and the query vector may be greater than a first threshold. For example, the similarity between the first type vector and the query vector may be less than a second threshold, and the second threshold may be greater than the first threshold.

[0078] In operation S23, the first processor 11 can prefetch the cluster to which the first type vector belongs among the multiple clusters of vectors in the vector database belongs to the second memory device 22.

[0079] In operation S24, the second processor 12 may search for a first-type cluster center among the cluster centers of the clusters to which the first-type vector found by the first processor 11 belongs. For example, the similarity between the first-type cluster center and the query vector is greater than a cluster center threshold. For example, the second processor 12 may perform a simple search based on the cluster centers of the clusters to which the first-type vector belongs. The second processor 12 may calculate the similarity between each cluster center of the clusters to which the first-type vector belongs and the query vector, and determine the cluster centers with similarity greater than the cluster center threshold as first-type cluster centers. In one or more examples, the similarity between a cluster center and a vector may be determined based on the distance between the cluster center and the vector.

[0080] In operation S25, the second processor 12 may determine the vector most similar to the query vector based on the cluster to which the center of a first type of cluster belongs among the clusters prefetched by the first processor 11 to the second memory device 22. For example, the second processor 12 may perform a cluster search based on the cluster to which the center of the first type of cluster belongs.

[0081] In operation S26, the second processor 12 can determine the inference result based on the vector most similar to the query vector.

[0082] In one or more examples, operation S23 performed by the first processor 11 and operation S24 performed by the second processor 12 may be performed simultaneously.

[0083] In one or more examples, the cluster center of the cluster to which the first type vector belongs is similar to the query vector. However, due to computational discrepancies between the first processor 11 and the second processor 12, there is a small probability that the cluster center of the cluster to which the first type vector belongs will have low similarity to the query vector (e.g., no first type cluster center is found). To avoid large data transfer and computational loads caused by not finding the first type cluster center, in this embodiment, when the first type cluster center is not found, the first processor 11 may perform a cluster search based on the cluster to which the first type vector belongs.

[0084] In the examples described above, the first processor 11 may look up a first type vector. To further reduce the amount of data transfer and computation caused by searching for the most similar vector to the query vector, the first processor 11 may look up a second type vector (the second type vector is more similar to the query vector than the first type vector) among multiple historical query vectors. In response to finding the second type vector, the first processor 11 may send at least one of the context of the second type vector pre-stored in the first memory device 21 and the KVcache to the second processor 12. In one or more examples, the first processor 11 may prefetch the cluster to which the first type vector belongs to the second memory device 22 in response to finding the first type vector among multiple historical query vectors and not finding the second type vector.

[0085] Figure 5 This is a flowchart illustrating a method of operating a first processor according to some example embodiments.

[0086] like Figure 5 As shown, in operation S410, the first processor 11 can receive a query and convert the query into a query vector.

[0087] In operation S420, the first processor 11 can obtain the indexes of multiple historical query vectors from the first memory device 21 and calculate the similarity between each historical query vector and the query vector to determine which of the first to third cases the query vector belongs to.

[0088] In one or more examples, the first case can represent that the query vector is relatively similar to the historical query vector. For example, when there are vectors of type 1 but no vectors of type 2 among multiple historical query vectors, the query vector can be relatively similar to the historical query vector.

[0089] In one or more examples, the second case can represent a query vector that is extremely similar to a historical query vector. For example, a query vector can be extremely similar to a historical query vector when a second type of vector exists among multiple historical query vectors.

[0090] In one or more examples, the third case can represent that the query vector is dissimilar to historical query vectors. For example, a query vector may be dissimilar to historical query vectors when there are no first-type vectors and second-type vectors among multiple historical query vectors.

[0091] In operation S430, the first processor 11 may search for a second type vector among multiple historical query vectors based on the calculated similarity (e.g., determine whether a second type vector exists).

[0092] When a second type vector exists, in operation S440, the first processor 11 can obtain the context and KVcache of the second type vector from the first memory device 21, and send the context and KVcache of the second type vector to the second processor 12, so that the second processor 12 can determine the inference result based on the context and KVcache of the second type vector.

[0093] When the second type vector does not exist, in operation S450, the first processor 11 can search for the first type vector among multiple historical query vectors, for example, to determine whether the first type vector exists.

[0094] When no vector of the first type exists, the above reference can be used. Figure 1 or Figure 2 The way the description is presented determines the result of the reasoning.

[0095] When a first type vector exists, in operation S460, the first processor 11 can send the identifier of the first type vector to the second processor 12, so that the second processor 12 can confirm whether the historical query vectors stored in the second memory device 22 include all the first type vectors indicated by the identifier.

[0096] When the first processor 11 determines, based on the response of the second processor 12, that all first type vectors indicated by the identifier are included in the historical query vectors stored in the second memory device 22, in operation S470, the first processor 11 may prefetch the cluster to which the first type vector belongs to the second memory device 22.

[0097] For example, the first processor 11 may prefetch a third type vector from the cluster to which the first type vector belongs from the first memory device 21 to the second memory device 22, and prefetch a fourth type vector from the cluster to which the first type vector belongs from the third memory device to the second memory device 22. For example, the third type vector is a vector whose usage frequency is greater than a first usage frequency threshold, and the fourth type vector is a vector whose usage frequency is less than the first usage frequency threshold. In one or more examples, the first processor 11 may also prefetch a fourth type vector from the cluster to which the first type vector belongs from the third memory device to the first memory device 21.

[0098] When the first processor 11 determines, based on the response of the second processor 12, that at least one of the first type vectors indicated by the identifier is not included in the historical query vectors stored in the second memory device 22, in operation S480, the first processor 11 may obtain the index of at least one first type vector from the first memory device 21 and send the index of at least one first type vector to the second processor 12. Then, the first processor 11 may execute operation S470.

[0099] When the second processor 12 determines that a first type of cluster center exists, after the second processor 12 performs a cluster search based on the cluster to which the first type of cluster center belongs and determines the vector most similar to the query vector, in operation S490, the first processor 11 may receive the vector most similar to the query vector (or the index of the vector most similar to the query vector, which includes the vector most similar to the query vector, the cluster center of the cluster to which the vector most similar to the query vector belongs, and the cluster number) from the second processor 12, obtain at least one of the context and KVcache of the vector most similar to the query vector from the first memory device 21 and / or the third memory device, and send at least one of the context and KVcache of the vector most similar to the query vector to the second processor 12.

[0100] In one or more examples, the first processor 11 may retrieve a first type of context and KVcache (e.g., hot context and KVcache) from the context and KVcache of the vector most similar to the query vector from the first memory device 21, and may retrieve a second type of context and KVcache (e.g., cold context and KVcache) from the context and KVcache of the vector most similar to the query vector from the third memory device. The first type of context and KVcache may represent context and KVcache with a usage frequency greater than a second usage frequency threshold, and the second type of context and KVcache may represent context and KVcache with a usage frequency less than the second usage frequency threshold.

[0101] In one or more examples, when the KVcache of the vector most similar to the query vector is not stored in the first memory device 21 and the third memory device, the first processor 11 may send the context of the vector most similar to the query vector to the second processor 12, which may (e.g., during LLM execution) compute the KVcache of the vector most similar to the query vector based on the context of the vector most similar to the query vector and send the KVcache of the vector most similar to the query vector to the first processor 11.

[0102] In one or more examples, when the KVcache of the vector most similar to the query vector is stored in the first memory device 21 and / or the third memory device, the first processor 11 may obtain the context and KVcache of the vector most similar to the query vector from the first memory device 21 and / or the third memory device, and send the context and KVcache of the vector most similar to the query vector to the second processor 12.

[0103] When no first-type cluster center exists, in operation S491, the first processor 11 may perform a cluster search based on the cluster to which the first-type vector belongs to determine the vector most similar to the query vector, obtain at least one of the context and KVcache of the vector most similar to the query vector from the first memory device 21 and / or the third memory device, and send at least one of the context and KVcache of the vector most similar to the query vector to the second processor 12. For example, the first processor 11 may determine the vector most similar to the query vector based on a third-type vector stored in the first memory device 21 within the cluster to which the first-type vector belongs, and a fourth-type vector prefetched from the third memory device to the first memory device 21 within the cluster to which the first-type vector belongs.

[0104] In operation S492, the first processor 11 can store the cluster number of the cluster to which the most similar vector to the query vector belongs, the cluster center of the cluster to which the most similar vector to the query vector belongs, and the query vector as the index of the query vector in the first memory device 21. It can obtain the context and KVcache of the most similar vector to the query vector, and store the context and KVcache of the most similar vector to the query vector as the context and KVcache of the query vector in the first memory device 21.

[0105] For example, when the KVcache of the vector most similar to the query vector is not stored in the first memory device 21 and the third memory device, the first processor 11 can receive the KVcache calculated by the second processor 12 based on the context of the vector most similar to the query vector from the second processor 12, and obtain the context of the vector most similar to the query vector from the first memory device 21 and / or the third memory device. As another example, when the KVcache of the vector most similar to the query vector is stored in the first memory device 21 and / or the third memory device, the first processor 11 can obtain the context and KVcache of the vector most similar to the query vector from the first memory device 21 and / or the third memory device.

[0106] In operation S493, the first processor 11 may store (or update) the indexes of a first predetermined number of historical query vectors with the highest usage frequency among a plurality of historical query vectors stored in the first memory device 21 from the first memory device 21 to the second memory device 22 via the second processor 12. In one or more examples, the first processor 11 may periodically update the historical query vectors stored in the second memory device 22.

[0107] In this embodiment, by storing frequently used historical query vectors and their respective cluster centers in a second memory device, the number of cluster centers that need to be transferred for simple searches can be reduced.

[0108] Figure 6 This is a flowchart illustrating a method for operating a second processor in a first case according to some example embodiments.

[0109] like Figure 6 As shown, in operation S510, the second processor 12 may receive the identifier of the first type vector from the first processor 11.

[0110] In operation S520, the second processor 12 can obtain the index of a first predetermined number of historical query vectors from the second memory device 22, and can confirm whether the historical query vectors stored in the second memory device 22 include all the first type vectors indicated by the identifier based on the received identifier and the index of the first predetermined number of historical query vectors.

[0111] When the historical query vector stored in the second memory device 22 includes all first-type vectors indicated by the identifier, in operation S530, the second processor 12 can obtain the index of all first-type vectors from the second memory device 22 based on the received identifier.

[0112] When at least one of the first type vectors indicated by the identifier is not included in the historical query vectors stored in the second memory device 22, in operation S540, the second processor 12 may obtain the index of at least one first type vector from the first memory device 21 via the first processor 11, and obtain from the second memory device 22 the index of the first type vector among the first type vectors indicated by the identifier received from the first processor 11 that exists in the historical query vectors stored in the second memory device 22.

[0113] However, embodiments of this disclosure are not limited to these configurations. For example, the index of a first-type vector or the cluster center of its constituent cluster can be obtained in other ways. For instance, the first processor 11 can send the indices of all first-type vectors indicated by identifiers directly from the first memory device 21 to the second processor 12.

[0114] In operation S550, the second processor 12 can search for the first type cluster center in the cluster centers in the index including the first type vector (e.g., determine whether the first type cluster center exists).

[0115] When a first type cluster center exists, in operation S560, the second processor 12 can obtain the cluster to which the first type cluster center belongs from the cluster to which the first type vector belongs in the prefetched first type vector of the second memory device 22 belongs based on the cluster number in the index of the first type vector corresponding to the first type cluster center, determine the vector most similar to the query vector based on the cluster to which the first type cluster center belongs, and send the vector most similar to the query vector (or the index of the vector most similar to the query vector) to the first processor 11.

[0116] When no first type cluster center exists, in operation S570, the second processor 12 can send the result indicating that no first type cluster center exists to the first processor 11, and wait for the first processor 11 to determine the vector most similar to the query vector.

[0117] In operation S580, the first processor 11 may obtain at least one of the context and KVcache of the vector most similar to the query vector from the first memory device 21 and / or the third memory device, and send the context and KVcache of the vector most similar to the query vector to the second processor 12. The second processor 12 may perform LLM based on the context and KVcache of the vector most similar to the query vector to determine the inference result.

[0118] When the KVcache of the vector most similar to the query vector is not stored in the first memory device 21 and the third memory device, in operation S590, the second processor 12 may store the KVcache calculated based on the context of the vector most similar to the query vector in the first memory device 21 via the first processor 11.

[0119] In this embodiment, when the second processor fails to find a cluster center similar to the query vector, the first processor can perform a cluster search, thereby reducing the waiting time required to transmit clusters and further improving the efficiency of inference.

[0120] Figure 7 This is a flowchart illustrating a method for operating a second processor in a second case according to some example embodiments.

[0121] like Figure 7 As shown, in operation S610, the second processor 12 can receive the context and KVcache of the second type vector from the first memory device 21 via the first processor 11.

[0122] In operation S620, the second processor 12 can perform LLM based on the context of the second type vector and KVcache to determine the inference result.

[0123] Figure 8 This is a diagram illustrating a first memory device according to some example embodiments.

[0124] like Figure 8 As shown, the first memory device 21 may include a first region 211, a second region 212, a third region 213, a fourth region 214, and a fifth region 215.

[0125] According to one or more example embodiments, the first region 211 may store indexes of multiple historical query vectors corresponding to multiple historical queries.

[0126] For example, multiple historical queries can be queries entered by the user within a predetermined time period in the past, starting from the current time.

[0127] For example, the index of each historical query vector includes the corresponding historical query vector, the cluster number of the cluster to which the historical query vector belongs, and the cluster center of the cluster to which the historical query vector belongs. For example, the cluster to which the historical query vector belongs could be the cluster to which the vector in the vector database most similar to the historical query vector belongs.

[0128] According to one or more example embodiments, the second region 212 may store the context and KVcache of each historical query vector (e.g., the context and KVcache of the vector in the vector database that is most similar to the historical query vector).

[0129] According to one or more example embodiments, the third region 213 may store a third type vector (or an index of a third type vector) among the vectors in the vector database. For example, the third type vector is a vector whose usage frequency is greater than a first usage frequency threshold.

[0130] According to one or more example embodiments, the fourth region 214 may temporarily store a fourth type vector (or an index of the fourth type vector) prefetched from a third memory device within the cluster to which the first type vector belongs. For example, the fourth type vector is a vector whose usage frequency is less than a first usage frequency threshold.

[0131] In one or more examples, each of the indices of the third-type vector and the fourth-type vector may include the corresponding vector, the cluster number of the cluster to which the corresponding vector belongs, and the cluster center of the cluster to which the corresponding vector belongs.

[0132] According to one or more example embodiments, the fifth region 215 may store the context of vectors in the vector database and the context and KVcache in which the frequency of use is greater than a second frequency threshold.

[0133] According to one or more example embodiments, the more times the context and KVcache of a historical query vector are reused, the higher the frequency of use of the historical query vector; the more times a vector in the vector database is compared with a query, the higher the frequency of use of the vector in the vector database; the more times the context and KVcache of a vector in the vector database are reused, the higher the frequency of use of the context and KVcache of the vector in the vector database.

[0134] Although Figure 8 Five regions are shown, and the five regions are separated from each other, but the example is not limited to this, and the first memory device 21 may include more or fewer regions, and each region may store at least one of the contents of the first region 211 to the fifth region 215 described above.

[0135] In one or more examples, the first memory device 21 may be memory that can be directly accessed by the first processor 11 (e.g., but not limited to, dynamic random access memory (DRAM)).

[0136] Figure 9 This is a diagram illustrating a second memory device according to some example embodiments.

[0137] like Figure 9 As shown, the second memory device 22 may include a first region 221, a second region 222, and a third region 223.

[0138] According to one or more example embodiments, the first region 221 may store indexes of a first predetermined number of historical query vectors that are most frequently used among a plurality of historical query vectors. For example, the index of each historical query vector includes the corresponding historical query vector, the cluster number of the cluster to which the corresponding historical query vector belongs, and the cluster center of the cluster to which the corresponding historical query vector belongs. For example, the index of historical query vectors stored in the first region 221 may be a part of the index of historical query vectors stored in the first memory device 21.

[0139] According to one or more example embodiments, the second region 222 may store the cluster to which the first type vector prefetched by the first processor 11 belongs.

[0140] According to one or more example embodiments, the third region 223 may store intermediate data generated during the execution of LLM.

[0141] Although Figure 9 The diagram shows three regions, which are separated from each other, but the example is not limited thereto, and the second memory device 22 may include more or fewer regions, and each region may store at least one of the contents of the first region 221 to the third region 223 described above.

[0142] In one or more examples, the second memory device 22 may be memory that can be directly accessed by the second processor 12 (e.g., but not limited to, high bandwidth memory (HBM)).

[0143] Figure 10 This is a diagram illustrating a third memory device according to some example embodiments.

[0144] According to one or more example embodiments, the inference system 1 may also include a third memory device.

[0145] like Figure 10 As shown, the third memory device 23 may include a first region 231 and a second region 232.

[0146] According to one or more example embodiments, the first region 231 may store a fourth type vector (or an index of a fourth type vector) among the vectors in the vector database. For example, a fourth type vector is a vector whose usage frequency is less than a first usage frequency threshold. For example, each index of a fourth type vector may include the corresponding vector, the cluster number of the cluster to which the corresponding vector belongs, and the cluster center of the cluster to which the corresponding vector belongs.

[0147] The second region 232 can store the context of vectors in the vector database and the context and KVcache in which the frequency of use is less than the second frequency threshold.

[0148] According to one or more example embodiments, the fourth type vector stored in the third memory device 23 and the third type vector stored in the first memory device 21 can constitute a vector in the vector database.

[0149] Although Figure 10 The example shows two regions that are separate from each other, but the example is not limited thereto, and the second memory device 22 may include more or fewer regions, and each region may store at least one of the contents of the first region 231 and the second region 232 described above.

[0150] In one or more examples, the third memory device 23 may be a memory expander (e.g., but not limited to, compute fast link (CXL) memory module DRAM (CMM-D)) that can be directly accessed by the first processor 11.

[0151] In this embodiment, by storing frequently used vectors, context, and KVcache in a first memory device and storing infrequently used vectors, context, and KVcache in a third memory device, the transmission time of frequently used vectors can be reduced, thereby further improving the efficiency of inference.

[0152] Figure 11 This is a schematic diagram illustrating how inference operations are performed in a first case according to some example embodiments.

[0153] exist Figure 11 For ease of understanding, assume that the first memory device (e.g., DRAM) 21 stores the indices I_A1 to I_An of the historical query vectors A1 to An corresponding to the historical queries a1 to an, and the second memory device (e.g., HBM) 22 stores the indices I_A1 and I_A2 of the historical query vectors A1 and A2, and the current input query is a.

[0154] In operation S11, the first processor (e.g., CPU) 11 can convert query a into query vector A.

[0155] In operation S12, the first processor 11 can read the indices I_A1 to I_An of the historical query vectors A1 to An from the first memory device 21.

[0156] In operation S13, the first processor 11 can calculate the similarity between each of the historical query vectors A1 to An and the query vector A.

[0157] In operation S14, the first processor 11 may send identifiers ID_A1 and ID_A3 of first type vectors A1 and A3 with a similarity greater than a first threshold and less than a second threshold with the query vector A.

[0158] In operation S15, the second processor (e.g., GPU) 12 can read the indices I_A1 and I_A2 of the two most frequently used historical query vectors A1 and A2 from the second memory device, and can determine that the historical query vectors stored in the second memory device 22 do not include the first type vector A3.

[0159] In operation S16, the second processor 12 may request the index I_A3 of the first type vector A3 from the first processor 11.

[0160] In operation S17, the first processor 11 can read the index I_A3 of the first type vector A3 from the first memory device 21 and send the index I_A3 of the first type vector A3 to the second processor 12.

[0161] In operation S18, the second processor 12 can use the indices I_A1 and I_A3 of the first type vectors A1 and A3 to determine whether a first type cluster center exists. For example, the second processor 12 can calculate the similarity between each of the cluster centers C_A1 and C_A3 in the indices I_A1 and I_A3 of the first type vectors A1 and A3 and the query vector A, and determine whether the similarity between each of the cluster centers C_A1 and C_A3 and the query vector A is greater than a cluster center threshold.

[0162] While performing operation S18, the first processor 11 can prefetch the clusters (e.g., C1 and C3, which may be the same or different) to which the first type vectors A1 and A3 belong to the second memory device.

[0163] For example, the first processor 11 may prefetch third-type vectors of clusters C1 and C3 from the first memory device 21 to the second memory device 22, and may prefetch fourth-type vectors of clusters C1 and C3 from the third memory device 23 to the second memory device 22. In one or more examples, the first processor 11 may also prefetch fourth-type vectors of clusters C1 and C3 from the third memory device 23 to the first memory device 21.

[0164] When a first type of cluster center exists (e.g., when the similarity between at least one of cluster centers C_A1 and C_A3 and query vector A is greater than the cluster center threshold), in operation S19, the second processor 12 may perform a cluster search.

[0165] For example, when the similarity between cluster center C_A1 and query vector A is greater than the cluster center threshold, the second processor 12 can determine the cluster number C1 in the index I_A1 of the first type vector A1 corresponding to cluster center C_A1, read the prefetched cluster C1 and cluster C1 in C3 from the second memory device 22, and calculate the similarity between each vector in cluster C1 and query vector A to determine the vector most similar to query vector A.

[0166] When no first-type cluster center exists, in operation S19, the first processor 11 may perform a cluster search. For example, the first processor 11 may read clusters C1 and C3 from the first memory device 21 and calculate the similarity between each vector in clusters C1 and C3 and the query vector A to determine the vector most similar to the query vector A.

[0167] After determining the vector most similar to query vector A, in operation S20, the second processor 12 may read at least one of the context and KVcache of the vector most similar to query vector A from the first memory device 21 and / or the third memory device 23 via the first processor 11, and perform LLM based on the context and at least one of KVcache of the vector most similar to query vector A to obtain the inference result.

[0168] In one or more examples, the second processor 12 may read the hot context and KVcache of the context and KVcache of the vector most similar to query vector A from the first memory device 21 via the first processor 11, and read the cold context and KVcache of the context and KVcache of the vector most similar to query vector A from the third memory device 23.

[0169] Figure 12 This is a schematic diagram illustrating how inference operations are performed in a second case according to some example embodiments.

[0170] like Figure 12 As shown, in operation S11, the first processor 11 can convert query a into query vector A.

[0171] In operation S12, the first processor 11 can read the indices I_A1 to I_An of the historical query vectors A1 to An from the first memory device 21.

[0172] In operation S13, the first processor 11 can calculate the similarity between each of the historical query vectors A1 to An and the query vector A, and can determine that the similarity between the historical query vector A1 and the query vector A is greater than a second threshold.

[0173] In operation S14', the first processor 11 can read the context and KVcache of the historical query vector A1 from the first memory device 21, and send the context and KVcache of the historical query vector A1 to the second processor 12.

[0174] In operation S15', the first processor 11 may perform LLM based on the context of the historical query vector A1 and KVcache to obtain inference results.

[0175] Figure 13 This is a flowchart illustrating the reasoning method according to some example embodiments in more detail.

[0176] like Figure 13 As shown, in operation S1310, the first processor 11 can receive a query and convert the query into a query vector.

[0177] In operation S1320, the first processor 11 may calculate the similarity between each of the historical query vectors stored in the first memory device 21 and the query vector.

[0178] In operation S1330, the first processor 11 may determine, based on the calculated similarity, whether a second type of vector exists among the historical query vectors stored in the first memory device 21.

[0179] When a second type vector exists, in operation S1331, the first processor 11 can obtain the context and KVcache of the second type vector from the first memory device 21 and send the context and KVcache of the second type vector to the second processor 12.

[0180] In operation S1332, the second processor 12 can perform LLM based on the context of the second type vector and KVcache.

[0181] When a second type vector does not exist, in operation S1340, the first processor 11 can determine whether a first type vector exists.

[0182] When no vector of the first type exists, in operation S1341, the above reference can be used. Figure 1 or Figure 2 The way the description is presented determines the result of the reasoning.

[0183] When a first type vector exists, in operation S1350, the first processor 11 can send the identifier of the first type vector to the second processor 12.

[0184] In operation S1360, the second processor 12 may confirm, based on the received identifier and the index of the first predetermined number of historical query vectors stored in the second memory device 22, whether the historical query vectors stored in the second memory device 22 include all the first type vectors indicated by the identifier.

[0185] When the historical query vector stored in the second memory device 22 includes all first-type vectors indicated by the identifier, in operation S1370, the second processor 12 can obtain the index of all first-type vectors from the second memory device 22 based on the received identifier.

[0186] When at least one of the first type vectors indicated by the identifier is not included in the historical query vectors stored in the second memory device 22, in operation S1380, the second processor 12 may obtain the index of at least one first type vector from the first memory device 21 via the first processor 11, and obtain from the second memory device 22 the index of the first type vector among the first type vectors indicated by the identifier received from the first processor 11 that exists in a first predetermined number of historical query vectors in the second memory device 22.

[0187] In operation S1390, the first processor 11 can prefetch the third type vector in the cluster to which the first type vector belongs from the first memory device 21 to the second memory device 22, and prefetch the fourth type vector in the cluster to which the first type vector belongs from the third memory device 23 to the first memory device 21 and the second memory device 22.

[0188] In operation S1391, the second processor 12 can determine whether a first type cluster center exists among the cluster centers in the index including the first type vector.

[0189] When a first type cluster center exists, in operation S1392, the second processor 12 can determine the vector most similar to the query vector based on the cluster to which the first type vector of the second memory device 22 belongs, and obtain at least one of the context and KVcache of the vector most similar to the query vector from the first memory device 21 and / or the third memory device 23 via the first processor 11.

[0190] When there is no first type cluster center, in operation S1393, the first processor 11 may perform a cluster search based on the cluster to which the first type vector belongs to determine the vector most similar to the query vector, and send at least one of the context of the vector most similar to the query vector and KVcache from the first memory device 21 and / or the third memory device 23 to the second processor 12.

[0191] In operation S1394, the second processor 12 may perform an LLM based on at least one of the context of the vector most similar to the query vector and the KVcache to determine the inference result. For example, the second processor 12 may input the context of the vector most similar to the query vector and at least one of the KVcache into the LLM to obtain the inference result from the LLM.

[0192] In operation S1395, the first processor 11 may store the cluster number of the cluster to which the most similar vector to the query vector belongs, the cluster center of the cluster to which the most similar vector to the query vector belongs, and the query vector as the index of the query vector in the first memory device 21, and store the context and KVcache of the most similar vector to the query vector as the context and KVcache of the query vector in the first memory device 21.

[0193] In operation S1396, the first processor 11 can periodically update the index of the first predetermined number of historical query vectors with the highest usage frequency among the multiple historical query vectors stored in the first memory device 21 from the first memory device 21 to the second memory device 22 via the second processor 12.

[0194] In one or more examples, when the second processor 12 performs computation on the Nth layer of a plurality of layers in an LLM, the first processor 11 may send at least one of the context for the (N+1)th layer and the KVcache to the second processor 12. For example, the Nth layer may represent any one of the plurality of layers.

[0195] In one or more examples, each time a second predetermined number of layers in a plurality of layers are computed, the second processor 12 may send the computation results of the second predetermined number of layers to the first processor.

[0196] In this embodiment, when performing the calculation of the Nth layer of the large language model, the data required for the N+1th layer is transmitted to reduce transmission waiting time and further improve the efficiency of inference.

[0197] In this embodiment, by storing intermediate data of as many layers as possible of the LLM, the operations of migrating intermediate data to and reading intermediate data from external devices can be reduced, thereby reducing IO frequency and IO overhead.

[0198] The devices, units, modules, and other components described herein are implemented by hardware components. Examples of hardware components that can be used to perform the operations described herein include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more hardware components performing the operations described herein are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements, such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. The hardware components implemented by the processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described herein. Hardware components may also access, manipulate, process, create, and store data pages in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. Hardware components may have any one or more different processing configurations, examples of which include: a single processor, a standalone processor, a parallel processor, single-instruction single-data-page (SISD) multiprocessing, single-instruction multiple-data-page (SIMD) multiprocessing, multiple-instruction single-data-page (MISD) multiprocessing, and multiple-instruction multiple-data-page (MIMD) multiprocessing.

[0199] The methods for performing the operations described in this application are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0200] Instructions or software for controlling a processor or computer to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure the processor or computer to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by the processor or computer. In another example, the instructions or software include high-level code that is executed by the processor or computer using an interpreter. Those skilled in the art can readily write instructions or software based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0201] Instructions or software used to control a processor or computer to implement hardware components and perform the methods described above, along with any associated data pages, data page files, and data page structures, are recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data page storage devices, optical data page storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data pages, data page files, and data page structures in a non-transitory manner and to provide instructions or software and any associated data pages, data page files, and data page structures to a processor or computer, enabling the processor or computer to execute the instructions.

[0202] Figure 14 yes Figure 3 A block diagram of example components of the inference system. Device 1400 may correspond to a user device and / or platform. Device 1400 may be any other suitable device (such as a TV, wall panel, etc.). Figure 14 As shown, device 1400 may include bus 1410, processor 1420, memory 1430, storage component 1440, input component 1450, output component 1460, and communication interface 1470.

[0203] Bus 1410 includes components that allow communication between components of device 1400. Processor 1420 is implemented in hardware, firmware, or a combination of hardware and software. Processor 1420 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processor (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other type of processing component. In some embodiments, processor 1420 includes one or more processors that can be programmed to perform functions. Memory 1430 includes random access memory (RAM), read-only memory (ROM), and / or other types of dynamic or static storage devices (e.g., flash memory, magnetic storage, and / or optical storage) that store information and / or instructions for use by processor 1420.

[0204] Storage component 1440 stores information and / or software related to the operation and use of device 1400. For example, storage component 1440 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state drives), laser discs (CDs), digital versatile discs (DVDs), floppy disks, cassette tapes, magnetic tapes, and / or other types of non-transitory computer-readable media, and corresponding drives.

[0205] Input component 1450 includes components that allow device 1400 to receive information, such as via user input (e.g., a touchscreen display, keyboard, keypad, mouse, button, switch, and / or microphone). Additionally or optionally, input component 1450 may include sensors for sensing information (e.g., a Global Positioning System (GPS) component, accelerometer, gyroscope, and / or actuator). Output component 1460 includes components that provide output information from device 1400 (e.g., a display, speaker, and / or one or more light-emitting diodes (LEDs)).

[0206] Communication interface 1470 includes transceiver-like components (e.g., transceiver and / or separate receiver and transmitter) that enable device 1400 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 1470 may allow device 1400 to receive information from other devices and / or provide information to other devices. For example, communication interface 1470 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0207] Apparatus 1400 may perform one or more of the processes described herein. Apparatus 1400 may perform these processes in response to processor 1420 executing software instructions stored in a non-transitory computer-readable medium, such as memory 1430 and / or storage component 1440. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0208] Software instructions may be read from another computer-readable medium or another device into memory 1430 and / or storage component 1440 via communication interface 1470. When the software instructions stored in memory 1430 and / or storage component 1440 are executed, processor 1420 may perform one or more processes described herein. Additionally or alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0209] Figure 14 The number and arrangement of components shown are provided for illustrative purposes only. In reality, with... Figure 14 Compared to the components shown, device 1400 may include additional components, fewer components, different components, or components arranged differently. Additionally or optionally, a set of components of device 1400 (e.g., one or more components) may perform one or more functions described as being performed by a further set of components of device 1400.

[0210] The first processor is further configured to: in response to finding a first-type vector among multiple historical query vectors and not finding a second-type vector, prefetch the cluster to which the first-type vector belongs to a second memory device; in response to finding a second-type vector among multiple historical query vectors, send the context and KVcache of the second-type vector pre-stored in the first memory device to the second processor; wherein the second processor is further configured to: determine the inference result based on the context and KVcache of the second-type vector, wherein the similarity between the second-type vector and the query vector is greater than a second threshold, and the second threshold is greater than a first threshold.

[0211] The first memory device is further configured to store indexes of multiple historical query vectors, and the first processor is further configured to store the indexes of a first predetermined number of historical query vectors with the highest frequency of use from the first memory device to the second memory device via the second processor, wherein each index in the indexes stored in the first memory device and the second memory device includes the corresponding vector and the cluster center of the cluster to which the corresponding vector belongs.

[0212] The first processor is further configured to send an identifier of a first type vector to a second processor, and the second processor is further configured to obtain the cluster center of the cluster to which the first type vector found by the first processor belongs by obtaining from a second memory device the index of a first type vector in the first type vector indicated by the identifier received from the first processor that exists in a first predetermined number of historical query vectors stored in the second memory device, and based on determining that at least one first type vector in the first type vector indicated by the identifier received from the first processor is not in the first predetermined number of historical query vectors stored in the second memory device, obtaining the index of at least one first type vector from the first memory device via the first processor.

[0213] Each index in the indexes stored in the first memory device and the second memory device also includes the cluster number of the cluster to which the corresponding vector belongs, wherein the second processor is further configured to: obtain the cluster to which the first type cluster center belongs from the second memory device based on the cluster number in the index of the first type vector corresponding to the first type cluster center.

[0214] The frequency of use of historical query vectors is directly proportional to the context of the historical query vectors and the number of times the KVcache is reused.

[0215] The second processor is also configured to: while the first processor prefetches the cluster to which the first type vector belongs to the second memory device, locate the center of the first type cluster.

[0216] The inference system also includes a third memory device configured to store a fourth type of vector among multiple clusters of vectors in a vector database, wherein the first memory device is further configured to store a third type of vector among multiple clusters of vectors in a vector database, wherein the third type of vector is a vector whose usage frequency is greater than a usage frequency threshold, and the fourth type of vector is a vector whose usage frequency is less than a usage frequency threshold.

[0217] The first processor is also configured to: prefetch a third type vector from the cluster to which the first type vector belongs from the first memory device to the second memory device, and prefetch a fourth type vector from the cluster to which the first type vector belongs from the third memory device to the second memory device.

[0218] The first processor is also configured to: prefetch the fourth type vector from the third memory device to the first memory device within the cluster to which the first type vector belongs; and when the center of the first type cluster is not found, determine the vector most similar to the query vector based on the third type vector stored in the first memory device within the cluster to which the first type vector belongs and the fourth type vector prefetched from the third memory device to the first memory device within the cluster to which the first type vector belongs.

[0219] The first memory device includes dynamic random access memory (DRAM), the second memory device includes high bandwidth memory (HBM), and the third memory device includes compute fast link (CXL) memory module DRAM.

[0220] The first processor is further configured to: receive from the second processor the vector most similar to the query vector, the cluster number of the cluster to which the vector most similar to the query vector belongs, and the cluster center of the cluster to which the vector most similar to the query vector belongs; obtain the context and KVcache of the vector most similar to the query vector; store the cluster number of the cluster to which the vector most similar to the query vector belongs, the cluster center of the cluster to which the vector most similar to the query vector belongs, and the query vector as an index of the query vector in a first memory device; and store the context and KVcache of the vector most similar to the query vector as the context and KVcache of the query vector in the first memory device.

[0221] The second processor is further configured to input at least one of the context of the vector most similar to the query vector and the KVcache into the large language model to obtain inference results from the large language model, wherein the first processor is further configured to send at least one of the context of the (N+1)th layer of the multiple layers of the large language model to the second processor when the second processor performs computation of the Nth layer of the multiple layers, wherein the Nth layer corresponds to one of the multiple layers.

[0222] The second processor is also configured to send the computation results of the second predetermined number of layers to the first processor in response to the execution of computation of the second predetermined number of layers among the multiple layers.

[0223] The first processor includes a central processing unit (CPU), and the second processor includes a graphics processing unit (GPU).

[0224] The step of prefetching the cluster to which a first-type vector belongs among multiple clusters of vectors in a vector database belongs to a second memory device includes: in response to finding a first-type vector among multiple historical query vectors and not finding a second-type vector, prefetching the cluster to which the first-type vector belongs to the second memory device. The inference method further includes: through a first processor, in response to finding a second-type vector among multiple historical query vectors, sending the context and KVcache of the second-type vector pre-stored in the first memory device to the second processor; through the second processor, determining the inference result based on the context and KVcache of the second-type vector, wherein the similarity between the second-type vector and the query vector is greater than a second threshold, and the second threshold is greater than a first threshold.

[0225] The inference method further includes: storing indexes of multiple historical query vectors through a first memory device; and storing the indexes of a first predetermined number of historical query vectors with the highest usage frequency from the first memory device to a second memory device via a second processor through a first processor, wherein each index in the indexes stored in the first memory device and the second memory device includes the corresponding vector and the cluster center of the cluster to which the corresponding vector belongs.

[0226] The reasoning method further includes: sending the identifier of the first type vector to the second processor via the first processor; obtaining the cluster center of the cluster to which the first type vector found by the first processor belongs via the second processor, wherein the step of obtaining the cluster center of the cluster to which the first type vector found by the first processor belongs further includes: obtaining from the second memory device the index of the first type vector in the first type vector indicated by the identifier received from the first processor that exists in a first predetermined number of historical query vectors stored in the second memory device, and obtaining the index of at least one first type vector from the first memory device via the first processor based on the fact that at least one first type vector in the first type vector indicated by the identifier received from the first processor is not in the first predetermined number of historical query vectors stored in the second memory device.

[0227] Each index stored in the first memory device and the second memory device also includes the cluster number of the cluster to which the corresponding vector belongs. The reasoning method further includes: using the second processor, obtaining the cluster to which the first type cluster center belongs from the second memory device based on the cluster number in the index of the first type vector corresponding to the first type cluster center.

[0228] The frequency of use of historical query vectors is directly proportional to the context of the historical query vectors and the number of times the KVcache is reused.

[0229] The step of finding the center of the first type of cluster also includes: finding the center of the first type of cluster while the first processor prefetches the cluster to which the first type vector belongs to the second memory device.

[0230] The inference method further includes: storing third-type vectors among multiple clusters of vectors in the vector database through a first memory device, and storing fourth-type vectors among multiple clusters of vectors in the vector database through a third memory device, wherein the third-type vectors are vectors whose usage frequency is greater than a usage frequency threshold, and the fourth-type vectors are vectors whose usage frequency is less than a usage frequency threshold.

[0231] The step of prefetching the cluster to which a first type vector belongs among multiple clusters of vectors in a vector database belongs to a second memory device includes: prefetching a third type vector from the first memory device to the second memory device from the first memory device, and prefetching a fourth type vector from the third memory device to the second memory device from the third memory device to the second memory device.

[0232] The reasoning method further includes: using the first processor, prefetching the fourth type vector from the third memory device to the first memory device within the cluster to which the first type vector belongs; when the center of the first type cluster is not found, determining the vector most similar to the query vector based on the third type vector stored in the first memory device within the cluster to which the first type vector belongs and the fourth type vector prefetched from the third memory device to the first memory device within the cluster to which the first type vector belongs.

[0233] The first memory device includes dynamic random access memory (DRAM), the second memory device includes high bandwidth memory (HBM), and the third memory device includes compute fast link (CXL) memory module DRAM.

[0234] The inference method further includes: receiving from the second processor the vector most similar to the query vector, the cluster number of the cluster to which the vector most similar to the query vector belongs, and the cluster center of the cluster to which the vector most similar to the query vector belongs; obtaining the context and KVcache of the vector most similar to the query vector; storing the cluster number of the cluster to which the vector most similar to the query vector belongs, the cluster center of the cluster to which the vector most similar to the query vector belongs, and the query vector as the index of the query vector in the first memory device; and storing the context and KVcache of the vector most similar to the query vector as the context and KVcache of the query vector in the first memory device.

[0235] The step of determining the inference result based on the vector most similar to the query vector further includes: inputting at least one of the context of the vector most similar to the query vector and KVcache into the large language model to obtain the inference result from the large language model. The inference method further includes: when the second processor performs the calculation of the Nth layer of the multiple layers of the large language model, the first processor sends at least one of the context of the (N+1)th layer of the multiple layers and KVcache to the second processor, wherein the Nth layer corresponds to one of the multiple layers.

[0236] The inference method further includes: in response to the execution of calculations for a second predetermined number of layers in a plurality of layers, sending the calculation results of the second predetermined number of layers to the first processor via the second processor.

[0237] The first processor includes a central processing unit (CPU), and the second processor includes a graphics processing unit (GPU).

[0238] Although this disclosure has been specifically shown and described with reference to exemplary embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A reasoning system, comprising: The first memory device is configured to store multiple historical query vectors corresponding to multiple historical queries; Second memory device; The first processor is configured as follows: Convert the query into a query vector. Search for a first type vector among the plurality of historical query vectors stored in the first memory device, and Prefetch the cluster to which the first type vector belongs from the multiple clusters of vectors in the vector database belongs to the second memory device; The second processor is configured as follows: Search for the first type cluster center within the cluster centers of the cluster to which the first type vector belongs, as found by the first processor. Based on the cluster to which the first type cluster center belongs among the clusters prefetched from the first processor to the second memory device, the vector most similar to the query vector is determined, and The inference result is determined based on the vector most similar to the query vector. Among them, the similarity between the first type vector and the query vector is greater than the first threshold. Among them, the similarity between the first type of cluster center and the query vector is greater than the cluster center threshold.

2. The reasoning system according to claim 1, in, The first processor is also configured as follows: In response to finding a first-type vector among the plurality of historical query vectors and not finding a second-type vector, the cluster to which the first-type vector belongs is prefetched into the second memory device. In response to finding a second type vector among the plurality of historical query vectors, the context and KVcache of the second type vector, pre-stored in the first memory device, are sent to the second processor. The second processor is further configured to determine the inference result based on the context of the second type vector and the KVcache. Among them, the similarity between the second type vector and the query vector is greater than the second threshold, and the second threshold is greater than the first threshold.

3. The reasoning system according to claim 1, in, The first memory device is also configured to store an index of the plurality of historical query vectors. The first processor is further configured to store the indexes of a first predetermined number of historical query vectors with the highest usage frequency among the plurality of historical query vectors from the first memory device into the second memory device via the second processor. Each index stored in the first memory device and the second memory device includes the corresponding vector and the cluster center of the cluster to which the corresponding vector belongs.

4. The reasoning system according to claim 3, in, The first processor is also configured to send the identifier of the first type vector to the second processor. The second processor is further configured to obtain the cluster center of the cluster to which the first type vector found by the first processor belongs by means of the following method: Retrieve from the second memory device the index of the first type vector in the first type vector indicated by the identifier received from the first processor, which exists in a first predetermined number of historical query vectors stored in the second memory device. Based on the determination that at least one of the first type vectors indicated by the identifier received from the first processor is not among a first predetermined number of historical query vectors stored in the second memory device, the index of the at least one first type vector is obtained from the first memory device via the first processor.

5. The reasoning system according to claim 3, wherein, Each index stored in the first memory device and the second memory device also includes the cluster number of the cluster to which the corresponding vector belongs. The second processor is further configured to: obtain the cluster to which the first type cluster center belongs from the second memory device based on the cluster number in the index of the first type vector corresponding to the first type cluster center.

6. The reasoning system according to claim 3, wherein, The frequency of use of historical query vectors is directly proportional to the context of the historical query vectors and the number of times the KVcache is reused.

7. The reasoning system according to claim 1, wherein, The second processor is also configured to: while the first processor prefetches the cluster to which the first type vector belongs to the second memory device, locate the center of the first type cluster.

8. The reasoning system according to claim 1, further comprising: The third memory device is configured to store a fourth type of vector among the plurality of clusters of vectors in the vector database. The first memory device is further configured to store a third type of vector among the plurality of clusters of vectors in the vector database. Among them, the third type of vector is the vector whose usage frequency is greater than the usage frequency threshold, and the fourth type of vector is the vector whose usage frequency is less than the usage frequency threshold.

9. The reasoning system according to claim 8, wherein, The first processor is also configured as follows: The third type vectors within the cluster to which the first type vectors belong are prefetched from the first memory device to the second memory device, and The fourth type vector in the cluster to which the first type vector belongs is prefetched from the third memory device to the second memory device.

10. The reasoning system according to claim 8, wherein, The first processor is also configured as follows: The fourth type vector in the cluster to which the first type vector belongs is prefetched from the third memory device to the first memory device. When the center of the first type cluster is not found, the vector most similar to the query vector is determined based on the third type vector stored in the first memory device in the cluster to which the first type vector belongs and the fourth type vector prefetched from the third memory device to the first memory device in the cluster to which the first type vector belongs.

11. The reasoning system according to claim 8, wherein, The first memory device includes dynamic random access memory (DRAM), the second memory device includes high-bandwidth memory, and the third memory device includes computing fast link memory module (DRAM).

12. The reasoning system according to claim 1, wherein, The first processor is also configured as follows: The second processor receives the vector most similar to the query vector, the cluster number of the cluster to which the vector most similar to the query vector belongs, and the cluster center of the cluster to which the vector most similar to the query vector belongs. Retrieve the context and KVcache of the vector most similar to the query vector. The cluster number of the cluster to which the most similar vector to the query vector belongs, the cluster center of the cluster to which the most similar vector to the query vector belongs, and the query vector are stored as the index of the query vector in the first memory device. The context of the most similar vector to the query vector and the KVcache are also stored as the context of the query vector and the KVcache in the first memory device.

13. The reasoning system according to claim 1, wherein, The second processor is also configured to input at least one of the context of the vector most similar to the query vector and the KVcache into the large language model to obtain inference results from the large language model. The first processor is further configured to: when the second processor performs computation on the Nth layer of the multiple layers of the large language model, send at least one of the context and KVcache for the (N+1)th layer of the multiple layers to the second processor. The Nth layer corresponds to one of the plurality of layers.

14. The reasoning system according to claim 13, wherein, The second processor is further configured to send the calculation results of the second predetermined number of layers to the first processor in response to the execution of calculations of the second predetermined number of layers among the plurality of layers.

15. The reasoning system according to claim 1, wherein, The first processor includes a central processing unit, and the second processor includes a graphics processing unit.

16. A reasoning method, comprising: The first memory device stores multiple historical query vectors corresponding to multiple historical queries. The first processor converts the query into a query vector; The first processor searches for a first type vector among the plurality of historical query vectors stored in the first memory device. The first processor prefetches the cluster to which the first type vector belongs among the multiple clusters of vectors in the vector database belongs to the second memory device. The second processor searches for the first type cluster center among the cluster centers of the clusters to which the first type vector found by the first processor belongs; The second processor determines the vector most similar to the query vector based on the cluster to which the first type cluster center belongs among the clusters prefetched by the first processor to the second memory device. The second processor determines the inference result based on the vector most similar to the query vector. Among them, the similarity between the first type vector and the query vector is greater than the first threshold. Among them, the similarity between the first type of cluster center and the query vector is greater than the cluster center threshold.

17. A non-transitory computer-readable storage medium storing a computer program, wherein, When the computer program is executed by the processor, the processor performs the reasoning method as described in claim 16.

Images