Embedded table accelerated construction method, embedded vector retrieval method, DSA and equipment

Abstract

The invention discloses an accelerated construction method of an embedded table, an embedded vector retrieval method, a DSA (Digital Subscriber Amplifier) and equipment. The accelerated construction method of the embedded table comprises the following steps: segmenting each embedded vector in N embedded vectors into M sub-vectors, and collecting the sub-vectors at the same segmentation position into the same sub-vector set to obtain M sub-vector sets; loading N sub-vectors in each sub-vector set into a plurality of matrix multiplication operation units, and carrying out parallel calculation on P clustering centers corresponding to each sub-vector set; loading each segmented embedded vector and the P * M clustering centers into a plurality of matrix multiplication operation units, and carrying out parallel calculation on the nearest clustering center of each sub-vector in each embedded vector; and constructing an index table matched with the N embedded vectors according to the nearest clustering center of each sub-vector in each embedded vector. The invention provides a method for constructing the embedded table adaptive to the hardware multi-thread capability of the DSA and retrieving the embedded vector, so that the retrieval speed is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of computer hardware, and in particular to an accelerated construction of an embedding table, an embedding vector retrieval method, a DSA and a device. Background Art

[0002] In the field of AI-Generated Content (AIGC), large-scale natural language models (LLM) have developed rapidly in recent years, and the scale of the models has continued to grow. The number of parameters in some models has reached hundreds of billions or even trillions. In addition, the amount of training data required to train the models has also increased, reaching trillions of tokens (the smallest unit of text processing).

[0003] During the reasoning process of a large model, the large model can obtain knowledge from an external knowledge base that is not training data, so that it can be used in a wider range of application scenarios. The storage of massive external knowledge requires the help of a vector database. External knowledge can be either structured data or unstructured data. All of this knowledge exists in the form of embedded vectors, forming a very large embedding table. The construction and retrieval speed of the embedding table often determines the performance of the vector database, thereby affecting the efficiency of model reasoning.

[0004] When querying approximate vectors from an embedding table, the existing technology compares the full data in the table, which results in slow query speed and obviously cannot meet the throughput and latency requirements in many scenarios. Therefore, how to efficiently construct an embedding table with a specific index structure to implement high-speed embedding vector retrieval algorithms on DSA (Domain Specific Architecture) devices is an important issue that needs to be solved. Summary of the invention

[0005] The embodiments of the present invention provide an accelerated construction of an embedded table, an embedded vector retrieval method, a DSA and a device, so as to provide a new method for constructing an embedded table and retrieving an embedded vector adapted to the hardware multi-threading capability of the DSA, thereby effectively improving the speed and efficiency of the retrieval.

[0006] According to one aspect of an embodiment of the present invention, a method for accelerating construction of an embedded table is provided, which is executed by a DSA, and the method includes:

[0007] Each of the N embedded vectors is divided into M sub-vectors, and the sub-vectors located at the same division position in each embedded vector are grouped into the same sub-vector set to obtain M sub-vector sets; N and M are integers greater than 1;

[0008] The N embedded vectors and M sub-vector sets after segmentation are stored in the DSA storage unit respectively;

[0009] N sub-vectors in each sub-vector set in the DSA storage unit are loaded into a plurality of matrix multiplication operation units in the DSA search engine, P cluster centers corresponding to each sub-vector set are calculated in parallel, and the calculated P*M cluster centers are stored in the DSA storage unit; P is an integer greater than 1;

[0010] Load each segmented embedding vector and P*M cluster centers in the DSA storage unit into multiple matrix multiplication units in the DSA search engine, and calculate the nearest cluster center of each subvector in each embedding vector in parallel;

[0011] According to the nearest cluster center of each subvector in each embedded vector, an index table matching the N embedded vectors is constructed and stored in the DSA storage unit, wherein each embedded vector in the index table corresponds to M index values ​​represented by the cluster center.

[0012] According to another aspect of an embodiment of the present invention, there is provided an embedding vector retrieval method, which is performed by a DSA, and the method includes:

[0013] Obtain the embedded vector to be searched, and after the embedded vector to be searched is divided into M search sub-vectors by a division unit in the DSA search engine, each search sub-vector is distributed to a plurality of matrix multiplication operation units in the DSA search engine by a movement unit in the DSA search engine;

[0014] Load the P*M cluster centers corresponding to the N embedded vectors stored in the DSA storage unit into multiple matrix multiplication units in the DSA search engine, and calculate the distance table corresponding to the embedded vector to be retrieved in parallel, where the distance table contains P*M distance values;

[0015] Loading the index table stored in the DSA storage unit and constructed by the accelerated construction method of the embedding table as described in any one of the embodiments of the present invention into multiple matrix multiplication operation units, and calculating the distance and value between each embedded vector and the embedded vector to be retrieved in parallel;

[0016] The N distances and values ​​corresponding to the N embedded vectors are loaded into the topk solution unit in the DSA search engine, and the K embedded vectors with the smallest distances to the embedded vector to be retrieved are calculated as the retrieval results for the embedded vector to be retrieved, where K is an integer greater than or equal to 1.

[0017] According to another aspect of an embodiment of the present invention, there is also provided a DSA, comprising: an off-chip storage unit, a storage controller, and a search engine;

[0018] The storage engine includes an on-chip storage unit, an address calculation unit, an on-chip register unit, a plurality of matrix multiplication units, a segmentation unit, a movement unit, a topk solving unit and a cache unit;

[0019] Among them, the units in the DSA architecture cooperate together to implement the accelerated construction method of the embedded table as described in any one of the embodiments of the present invention, or implement the embedded vector retrieval method as described in any one of the embodiments of the present invention.

[0020] According to another aspect of the embodiments of the present invention, an electronic device is provided, and the electronic device includes the DSA as described in any one of the embodiments of the present invention.

[0021] The technical solution of the embodiment of the present invention realizes a method for constructing an embedding table of a specific structure and retrieving an embedding vector, and runs on an electronic device adapted to a specific DSA, which can greatly improve the efficiency of constructing the embedding table and retrieving the embedding vector. Compared with general computing units such as CPU (Central Processing Unit), the embodiments of the present invention choose to use DSA to construct the embedding table and retrieve the embedding vector, which can be well matched to the process of comparing and searching the embedded vector, and by leveraging the hardware multi-threading capability of DSA, it can further achieve parallel acceleration, thereby effectively improving the retrieval speed and efficiency of the embedded vector, and then in the process of model fine-tuning or model reasoning of a large language model, it can effectively improve the retrieval speed of the external knowledge vector, and correspondingly improve the speed of model fine-tuning or reasoning.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a method for accelerating construction of an embedded table provided according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of a process of constructing an embedded table applicable to an embodiment of the present invention;

[0026] Figure 3is a flow chart of an embedded vector retrieval method provided according to an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of an embedded table retrieval process applicable to an embodiment of the present invention;

[0028] Figure 5 is a hardware architecture diagram of a DSA provided by an embodiment of the present invention;

[0029] Figure 6 It is a schematic diagram of a specific architecture of a search engine in a DSA provided by an embodiment of the present invention;

[0030] Figure 7 is a schematic diagram of a distance calculation process based on DSA applicable to an embodiment of the present invention;

[0031] Figure 8 It is a schematic diagram of a distance search and summation process based on DSA applicable to an embodiment of the present invention.

[0032] Fig. 9 It is a structural diagram of an electronic device to which an embodiment of the present invention is applicable. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Figure 1A flowchart of an accelerated construction method of an embedding table provided in an embodiment of the present invention is applicable to constructing an embedding table of a specific format based on a large number of embedding vectors to efficiently respond to the retrieval of a specific embedding vector. The method can be executed by a DSA of a specific architecture, which can generally be configured in an electronic device. Specifically, the DSA can be understood as a hardware design optimized for a specific application or workload.

[0036] Correspondingly, such as Figure 1 As shown, the method includes:

[0037] S110, each of the N embedded vectors is divided into M sub-vectors, and each sub-vector located at the same division position in each embedded vector is grouped into the same sub-vector set to obtain M sub-vector sets, where N and M are both integers greater than 1.

[0038] Among them, the above-mentioned embedding vector can be a vector representation of knowledge data. Specifically, the embedding vector can be a vector representation of external knowledge used by a large language model in the process of model fine-tuning or model reasoning.

[0039] In this embodiment, taking the construction of a corresponding index table based on N pieces of knowledge embedding vectors as an example, the construction process of the entire index table is described.

[0040] Specifically, for each embedding vector, it is first divided into M sub-vectors, that is, each embedding vector is represented by M sub-vectors, and then N embedding vectors have a total of N*M sub-vectors.

[0041] In a specific example, embedding vector 1 is split into {sub-vector 11, sub-vector 12, ..., sub-vector 1M}, embedding vector 2 is split into {sub-vector 21, sub-vector 22, ..., sub-vector 2M}, and so on, and embedding vector N is split into {sub-vector N1, sub-vector N2, ..., sub-vector NM}.

[0042] At this time, the above-mentioned sub-vector 11, sub-vector 21, …, sub-vector N1 can be understood as sub-vectors located at the same cut position, and further, sub-vector 11, sub-vector 21, …, sub-vector N1 can be grouped into sub-vector set 1 to obtain sub-vector set 1 {sub-vector 11, sub-vector 21, …, sub-vector N1}, and correspondingly obtain sub-vector set 2 {sub-vector 12, sub-vector 22, …, sub-vector N2}, …, sub-vector set M {sub-vector 1M, sub-vector 2M, …, sub-vector NM}, a total of M sub-vector sets.

[0043] In this embodiment, the above-mentioned N and M can be preset according to actual conditions. For example, they can be determined based on parameters such as the total number of embedded vectors for constructing the embedding table, the feature dimensions (or vector length) contained in the embedded vectors, and the data processing capability of the DSA. This embodiment does not limit this.

[0044] Optionally, the operation of dividing each of the N embedded vectors into M sub-vectors can be performed by a dedicated hardware dividing unit in the DSA to increase the dividing speed of the embedded vectors.

[0045] S120, storing the segmented N embedding vectors and M sub-vector sets in DSA storage units respectively.

[0046] In this embodiment, considering the limited number of computing units on the DSA, it may not be possible to establish the table entry content of the corresponding embedding table for all N embedding vectors at one time. Therefore, the N*M sub-vectors of the N embedded vectors after segmentation and the M sub-vector sets can be first stored in the storage unit of the DSA (that is, the DSA storage unit). Afterwards, each time a portion of the data items is taken out, the computing unit on the DSA performs a single data processing to realize the local construction process of the embedding table.

[0047] Specifically, the DSA storage unit may be an on-chip storage unit of the DSA, or an off-chip storage unit adapted to the DSA, which is not limited in this embodiment.

[0048] S130, loading the N sub-vectors in each sub-vector set in the DSA storage unit into a plurality of matrix multiplication operation units in the DSA search engine, calculating P cluster centers corresponding to each sub-vector set in parallel, and storing the calculated P*M cluster centers in the DSA storage unit, where P is an integer greater than 1.

[0049] In the DSA of this embodiment, a search engine (i.e., DSA search engine) is provided, and in the DSA search engine, a plurality of matrix multiplication operation units are provided. The matrix multiplication operation unit can be understood as an operation unit dedicated to batch accelerated distance calculation, which is a computing hardware specially designed by the embodiments of the present invention to realize the accelerated construction process of the embedded table.

[0050] Among them, when loading the N sub-vectors in each sub-vector set in the DSA storage unit into multiple matrix multiplication operation units in the DSA search engine, you can choose to load the N sub-vectors in one sub-vector set each time for calculation, or you can also choose to load the N sub-vectors in two or more sub-vector sets each time for calculation. Technical personnel in this field can preset it according to the number of matrix multiplication operation units in the DSA search engine and the actual application computing requirements.

[0051] In an optional implementation of this embodiment, N sub-vectors in each sub-vector set in the DSA storage unit are loaded into multiple matrix multiplication operation units in the DSA search engine, and P cluster centers corresponding to each sub-vector set are calculated in parallel, which may specifically include:

[0052] Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit:

[0053] Loading N target subvectors from the target subvector set in the DSA storage unit;

[0054] Based on a preset clustering algorithm, P cluster centers matching the target sub-vector set are calculated through multiple matrix multiplication operations.

[0055] In this optional implementation, each matrix multiplication operation unit can be used alone to calculate P cluster centers corresponding to the subvector set according to the N subvectors contained in the same subvector set. Through the above settings, P cluster centers corresponding to each subvector set can be calculated, and then, for M subvector sets, a total of P*M cluster centers can be calculated.

[0056] Specifically, each P cluster centers has a corresponding sub-vector set. For example, sub-vector set 1 corresponds to cluster center set 1. The P cluster centers in cluster center set 1 are obtained after clustering calculation using sub-vector 11, sub-vector 21, ..., sub-vector N1 in cluster center set 1.

[0057] S140, loading each segmented embedding vector and P*M cluster centers in the DSA storage unit into multiple matrix multiplication operation units in the DSA search engine, and calculating the nearest cluster center of each subvector in each embedding vector in parallel.

[0058] It can be understood that the subvectors at the same cut position in the N embedded vectors are grouped in the same subvector set, and each subvector set can correspond to P cluster centers. Furthermore, the subvectors at different cut positions in an embedded vector can find a nearest cluster center among the P cluster centers of the subvector set to which they belong, and further, each subvector in the embedded vector can be represented by the matching nearest cluster center.

[0059] In an optional implementation of this embodiment, each segmented embedding vector and P*M cluster centers in the DSA storage unit are loaded into multiple matrix multiplication operation units in the DSA search engine, and the nearest cluster center of each subvector in each embedding vector is calculated in parallel, which may include:

[0060] Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit:

[0061] Load a target sub-vector in the target embedding vector after segmentation in the DSA storage unit, and load P target cluster centers corresponding to the sub-vector set to which the target sub-vector belongs among the P*M cluster centers;

[0062] The distance values ​​between the one target sub-vector and the P target cluster centers are calculated respectively by using a distance calculation formula, and the target cluster center with the smallest distance value is determined as the nearest cluster center corresponding to the one target sub-vector.

[0063] In a specific example, for subvector 11 in embedding vector 1, the nearest cluster center with the smallest distance value thereto can be obtained from cluster center set 1 corresponding to subvector set 1.

[0064] In this embodiment, each matrix multiplication operation unit can be used to determine the nearest cluster center for a subvector of a certain embedded vector. It can be understood that when each matrix multiplication operation unit performs the above calculation process, it has no data dependence on the calculation results of other matrix multiplication operation units. Therefore, multiple matrix multiplication operation units can perform calculations completely in parallel to finally calculate the nearest cluster center of each subvector in each embedded vector.

[0065] S150: construct an index table matching the N embedded vectors according to the nearest cluster center of each subvector in each embedded vector, and store the index table in the DSA storage unit.

[0066] In the index table, each embedded vector corresponds to M index values ​​represented by the cluster center.

[0067] In this embodiment, when constructing an index table for N embedded vectors, a total of N index records can be constructed in the index table, each embedded vector corresponds to an index record, and each index record includes M index values, each index value corresponds to a cluster center.

[0068] Similarly, the DSA storage unit may also be an on-chip or off-chip storage unit of the DSA, preferably an on-chip storage unit of the DSA.

[0069] The technical solution of the embodiment of the present invention realizes a method for constructing an embedding table of a specific structure, and runs on an electronic device adapted to a specific DSA, which can greatly improve the efficiency of constructing the embedding table. Compared with general computing units such as CPU, the embodiments of the present invention choose to use DSA to implement the construction process of the embedding table, which can be well matched with the process of comparing and searching the embedding vector, and by leveraging the hardware multi-threading capability of DSA, it can further achieve parallel acceleration, thereby effectively improving the construction speed and efficiency of the embedding table. The embedding table based on the above-mentioned specific structure can effectively improve the retrieval speed and efficiency of the embedding vector, and then in the process of model fine-tuning or model reasoning of a large language model, it can effectively improve the retrieval speed of the external knowledge vector, and correspondingly improve the speed of model fine-tuning or reasoning.

[0070] Specifically, in Figure 2 A schematic diagram of an embedded table construction process applicable to an embodiment of the present invention is shown in FIG. Figure 2 As shown, when constructing an embedding table of a specific structure, the method of the embodiment of the present invention constructs the data of the embedding table as N rows of embedding vectors, and each row is an embedding vector of length D. Furthermore, the embedding table includes N rows of index representations for the N rows of embedding vectors. The specific construction process of the embedding table is as follows:

[0071] 1. Load part or all of the embedding table data (N rows of embedded vectors) into the storage unit of the DSA. Each embedded vector is equally divided into M sub-vectors, and all vectors are divided into M vector subsets.

[0072] 2. Traverse the N rows of sub-vectors of each subset, load them into the matrix multiplication unit of DSA for vector calculation, accelerate the vector calculation process, and use a clustering algorithm to find P cluster centers. M subsets get a total of P×M cluster centers;

[0073] 3. Traverse each subvector of each embedded vector, load it into the matrix multiplication unit of DSA for distance calculation, and find the nearest cluster center from the P cluster centers of its subset by distance calculation between vectors, and retain the position index value of the center. Each vector obtains M index values, and all original vectors (N rows of embedded vectors) can be represented by N×M index values.

[0074] It should be further emphasized that, in the process of constructing the embedding table of the specific structure, the method of the embodiment of the present invention divides the originally longer D-dimensional vector into M subsets, and finally represents them by shorter M-dimensional vectors, which can save the storage space occupied by the embedded vector to a great extent, and the short vector will also speed up the subsequent search and retrieval process of the embedded table.

[0075] Figure 3This is a flowchart of an embedding vector retrieval method provided by an embodiment of the present invention. This embodiment is applicable to the case of quickly retrieving and obtaining the required embedding vector in a pre-built embedding table of a specific structure. The method can be executed by a DSA of a specific architecture, which can generally be configured in an electronic device. Figure 3 As shown, the method includes:

[0076] S310, obtaining the embedded vector to be retrieved, and after splitting the embedded vector to be retrieved into M retrieval sub-vectors through the splitting unit in the DSA search engine, distributing each retrieval sub-vector to multiple matrix multiplication operation units in the DSA search engine through the moving unit in the DSA search engine.

[0077] In a scenario where a large language model is fine-tuned or inferred in a manner applicable to the present embodiment, it may be necessary to input a set knowledge vector into the large language model to assist the large language model in subsequent fine-tuning or calculation processes. At this time, a vector similar to the required knowledge vector may be first obtained, that is, the embedding vector to be retrieved, and then, in the aforementioned index table constructed based on each knowledge vector, the index of one or more embedding vectors that meet the similarity condition with the embedding vector to be retrieved may be retrieved, and finally, the retrieved embedding vectors may be provided to the large language model for subsequent processing.

[0078] In this embodiment, since each embedded vector is divided into M sub-vectors before the index table stores the embedded vectors, in order to ensure the consistency of data retrieval, when obtaining the embedded vector to be retrieved, the embedded vector to be retrieved is also divided into M retrieval sub-vectors.

[0079] After obtaining the above M search sub-vectors, each search sub-vector can be distributed to multiple matrix multiplication units in the DSA search engine through a transfer unit in the DSA search engine. By using a hardware transfer unit, the data transfer function can be efficiently implemented.

[0080] S320, load the P*M cluster centers corresponding to the N embedded vectors stored in the DSA storage unit into multiple matrix multiplication operation units in the DSA search engine, and calculate in parallel the distance table corresponding to the embedded vector to be retrieved, wherein the distance table contains P*M distance values.

[0081] In this embodiment, the embedding vector to be retrieved is divided into M retrieval sub-vectors, each retrieval sub-vector actually corresponds to a sub-vector set used when constructing the embedding table, and further, each retrieval sub-vector actually corresponds to P cluster centers associated with a matched sub-vector set.

[0082] Continuing with the previous example, the retrieval subvector 1 of the embedded vector to be retrieved corresponds to the cluster center set 1 corresponding to the subvector set 1. Then, the distance values ​​between the retrieval subvector 1 and the P cluster centers in the cluster center set 1 can be calculated respectively.

[0083] Correspondingly, a total of P*M distance values ​​can be calculated from the M search sub-vectors.

[0084] In an optional implementation of this embodiment, the P*M cluster centers corresponding to the N embedded vectors stored in the DSA storage unit are loaded into multiple matrix multiplication operation units in the DSA search engine, and the distance table corresponding to the embedded vector to be retrieved is calculated in parallel, which may include:

[0085] Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit:

[0086] Among the P*M cluster centers stored in the DSA storage unit, load the P target cluster centers that match the target retrieval subvector distributed to this unit;

[0087] Calculating P distance values ​​between the target retrieval subvector and the P target cluster centers;

[0088] After obtaining P distance values ​​corresponding to each search sub-vector of the embedded vector to be searched, a distance table consisting of P*M distance values ​​is organized.

[0089] In this optional implementation, each matrix multiplication operation unit is used to independently calculate P distance values ​​between a search sub-vector and the associated P cluster centers. Since there is no data dependency between the calculation data used by each matrix multiplication operation unit, each matrix multiplication operation unit can be executed in parallel until P distance values ​​corresponding to each search sub-vector in the M search sub-vectors are calculated and organized to obtain a distance table.

[0090] S330: Load the index table stored in the DSA storage unit and constructed by the method described in any one of the embodiments of the present invention into multiple matrix multiplication operation units, and calculate the distance and value between each embedded vector and the embedded vector to be retrieved in parallel.

[0091] In this embodiment, based on the distance table calculated above and the pre-constructed index table of a specific structure, the distance and value between each embedded vector and the embedded vector to be retrieved can be obtained by searching the distance value in the distance table.

[0092] That is, in the index table, each embedding vector is described by M nearest cluster centers. By retrieving the distance value between the embedding vector to be retrieved and the M nearest cluster centers of each embedding vector in the above distance table, the distance and value between each embedding vector and the embedding vector to be retrieved can be obtained accordingly.

[0093] In an optional implementation of this embodiment, loading the index table stored in the DSA storage unit into multiple matrix multiplication operation units, and calculating the distance and value between each embedded vector and the embedded vector to be retrieved in parallel, may include:

[0094] Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit:

[0095] Loading M target index values ​​corresponding to the target embedding vector from the index table stored in the DSA storage unit;

[0096] Query the distance table according to the M target index values ​​to obtain M distance values ​​corresponding to the target embedding vector;

[0097] The M distance values ​​are summed to obtain the distance sum value corresponding to the target embedding vector.

[0098] S340, load the N distances and values ​​corresponding to the N embedded vectors into the topk solution unit in the DSA search engine, calculate the K embedded vectors with the smallest distances to the embedded vector to be retrieved, as the search results for the embedded vector to be retrieved, where K is an integer greater than or equal to 1.

[0099] In this embodiment, after obtaining N distances and values ​​corresponding to the embedded vector to be retrieved and N embedded vectors through parallel calculation by multiple matrix multiplication operation units, combined with a pre-built data recall strategy, for example, only obtaining the first K embedded vectors with the smallest distance and value, the hardware topk solution unit in the DSA search engine can be used to calculate the K embedded vectors with the smallest distance to the embedded vector to be retrieved as the retrieval result for the embedded vector to be retrieved.

[0100] In an optional implementation of this embodiment, N distance sum values ​​corresponding to N embedded vectors are loaded into a topk solving unit in a DSA search engine, and K embedded vectors having the smallest distance to the embedded vector to be retrieved are calculated, which may include:

[0101] The topk solving unit selects K distance values ​​with the smallest distances from the N distance sum values ​​based on a preset priority queue algorithm, and obtains K embedding vectors matching the K distance values.

[0102] It is understandable that the K embedding vectors closest to the embedding vector to be retrieved can be obtained from the above N embedding vectors by full sorting. However, the computational cost of this implementation is very high. Based on this, in each embodiment of the present invention, a topk solving unit in hardware form is developed separately to solve only the smallest K distance values ​​based on the priority queue algorithm. Through the above settings, the hardware acceleration capability in the embedding vector retrieval process is further expanded to effectively improve the retrieval speed of the embedding vector.

[0103] The technical solution of the embodiment of the present invention realizes a method for retrieving embedded vectors based on an embedded table of a specific structure, and runs on an electronic device adapted to a specific DSA, which can greatly improve the retrieval efficiency of embedded vectors. Compared with general computing units such as CPUs, the embodiments of the present invention choose to use DSA to implement the retrieval process of embedded vectors, which can be well matched to the process of matching and searching embedded vectors, and by leveraging the hardware multi-threading capability of DSA, it can further achieve parallel acceleration, thereby effectively improving the retrieval speed and efficiency of embedded vectors, and further in the process of model fine-tuning or model reasoning of large language models, effectively improve the retrieval speed of external knowledge vectors, and correspondingly improve the speed of model fine-tuning or reasoning.

[0104] Specifically, in Figure 4 A schematic diagram of an embedded table search process applicable to an embodiment of the present invention is shown in FIG. Figure 4 As shown, after the embedding table is constructed by the accelerated embedding table construction method provided by the embodiment of the present invention, a query vector Q (that is, the embedding vector to be retrieved) is given, and the process of retrieving the embedding table is as follows:

[0105] 1. Split the query vector Q into M sub-vectors (using the splitting unit);

[0106] 2. For each sub-vector of the query vector Q, load it into multiple matrix multiplication units in the DSA, and calculate a Euclidean distance between the sub-vector and the P cluster centers of the sub-vector set to which it belongs through vector calculation. A sub-vector set will obtain P distances, and a total of P×M distances are obtained to form a distance table, which is stored in the storage unit of the DSA;

[0107] 3. For the index table consisting of N×M index values ​​obtained in the construction process, each row vector has M index values. For each index value, the distance value of the corresponding index position is obtained from the P distance values ​​corresponding to the sub-vector set to which it belongs. A row vector obtains M distance values, and then the distance values ​​are summed to obtain the distance sum. N row vectors obtain a total of N distance sums;

[0108] 4. For the N distance sums, according to the priority queue algorithm, select the K distance values ​​with the smallest distance values ​​and the corresponding vector index values ​​(that is, the index value used to uniquely identify an embedded vector) as the search results for this time.

[0109] Further, in Figure 5 A schematic diagram of the hardware structure of a DSA provided by an embodiment of the present invention is shown in FIG. Figure 6 A specific architecture diagram of a search engine in a DSA provided by an embodiment of the present invention is shown in FIG.

[0110] like Figure 5 As shown, the DSA specifically includes: an off-chip storage unit 510, a storage controller 520 and a search engine 530 (search engine).

[0111] In a specific example, the off-chip storage unit 510 may be embodied as a plurality of independent DRAMs (Dynamic Random Access Memory). Figure 5 The example is 4. Furthermore, the number of memory controllers 520 (Memory controller) matches the number of DRAMs, and a communication connection is established through a bus.

[0112] Further, such as Figure 6 As shown, the search engine specifically includes an on-chip storage unit 610, an address calculation unit 620, an on-chip register unit 630, and a plurality of matrix multiplication operation units 640 ( Figure 6 Only two are used as examples), a splitting unit 650, a moving unit 660, a topk solving unit 670 and a cache unit 680.

[0113] Specifically, the on-chip storage unit 610 may be an L1 SRAM (Static Random-Access Memory), which represents the on-chip storage of the adjacent computing unit (i.e., the matrix multiplication operation unit 640); the address calculation unit 620 is a dedicated address calculation unit; the on-chip register unit 630 is used to implement the on-chip register function; the matrix multiplication operation unit 640 is an operation unit dedicated to batch accelerated distance calculation; the splitting unit 650 is an operation unit for fast vector splitting, and the moving unit 660 is a hardware unit for efficient data transfer; the topk solving unit 670 is a topk solving unit based on a priority queue algorithm, which can receive the batch distances calculated by the matrix multiplication operation unit 640 in real time (i.e., on the fly) and count the topk values, and the cache unit 680 may specifically be a first-in-first-out (FIFO) queue, which is used to cache the intermediate calculation results of the matrix multiplication operation unit 640.

[0114] Among them, the units in the DSA architecture cooperate together to implement the accelerated construction method of the embedded table as described in any one of the embodiments of the present invention, or implement the embedded vector retrieval method as described in any one of the embodiments of the present invention.

[0115] Specifically, the accelerated construction method of the embedded table as described in each embodiment of the present invention is run in DSA hardware, which can accelerate the process of calculating P×M cluster centers through a clustering algorithm, and the process of finding the nearest cluster center by calculating the distance between vectors.

[0116] Among them, Figure 7 A schematic diagram of a distance calculation process based on DSA implemented in accordance with an embodiment of the present invention is shown in FIG. Figure 7 As shown, in order to intuitively describe the distance calculation process, DSA is logically divided into DSA storage units ( Figure 7 The uppermost sub-frame in the ) and the DSA calculation unit ( Figure 7 (the portion located within the subframe defined by the two threads).

[0117] Specifically, multiple hardware execution threads can be pre-created inside the DSA, and a set number of vector calculation units (i.e., matrix multiplication operation units) can be allocated to each hardware execution thread. After that, each hardware execution thread is triggered in parallel, and each vector calculation unit in the hardware execution thread loads a sub-vector located in the vector subset and a set number of cluster centers from the DSA storage unit, and calculates the vector distance between the sub-vector and each cluster center, and determines the nearest cluster center based on the vector distance calculation result, or provides the calculated intermediate result to the DSA storage unit for storage.

[0118] Specifically, the embedded vector retrieval method as described in the embodiments of the present invention is run in DSA hardware, which can accelerate the process of calculating a Euclidean distance between each subvector of the query vector and the P cluster center vectors of the subset to which it belongs, and obtaining a total of P×M distance processes, and for the N×M index table obtained in the construction process, for each index value, the distance to the corresponding index position is taken from the P distances of the subset to which it belongs, and then the distance values ​​are summed, and a total of N distance sum calculation processes are obtained.

[0119] Among them, Figure 8 A schematic diagram of a distance search and summation process based on DSA applicable to an embodiment of the present invention is shown in FIG.

[0120] It needs to be emphasized again that in the process of constructing the embedding table and retrieving the embedding vector, it can be seen that the main objects of operation in many steps of the complete calculation process are vectors, and vectors are composed of a series of elements. In DSA hardware, its basic computing unit also has a large number of vector computing units (that is, matrix multiplication units), so many vectors can be loaded into these vector computing units at the same time, and multiple vector computing units can be executed simultaneously in one clock cycle, that is, the calculation actions of multiple vectors can be completed. In addition, there are a large number of hardware execution threads in this DSA hardware, which can further improve the ability of parallel computing and accelerate the calculation process.

[0121] That is, each embodiment of the present invention first designs a software calculation method, and the algorithm constructs an embedding table of a specific structure and quickly searches for a specified embedding vector from the embedding table to accelerate the construction and retrieval process of the embedding table, and can run in DSA hardware; in addition, a corresponding DSA dedicated chip architecture is designed to accelerate the entire algorithm flow, and the distance calculation, search, summation and other processes in the embedding table can be accelerated by using the special hardware structure of DSA and its vector calculation unit; finally, the software calculation method divides the data and utilizes the dedicated acceleration unit of DSA to cooperate with multi-threading to process different data blocks, so as to execute a series of operations in the algorithm in parallel, which greatly improves the execution speed compared to the serial execution of the CPU.

[0122] exist Fig. 9 FIG. 2 shows a structural diagram of an electronic device to which an embodiment of the present invention is applicable. Fig. 9 As shown, the electronic device may include one or more DSA910 as described in any embodiment of the present invention to implement the accelerated construction method of the embedding table as described in any embodiment of the present invention or the embedding vector retrieval method as described in any embodiment of the present invention.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for accelerating the construction of an embedded table, characterized in that: Executed by a domain specific architecture DSA, the method comprises: Each of the N embedded vectors is divided into M sub-vectors, and the sub-vectors located at the same division position in each embedded vector are grouped into the same sub-vector set to obtain M sub-vector sets; N and M are integers greater than 1; The N embedded vectors and M sub-vector sets after segmentation are stored in the DSA storage unit respectively; N sub-vectors in each sub-vector set in the DSA storage unit are loaded into a plurality of matrix multiplication operation units in the DSA search engine, P cluster centers corresponding to each sub-vector set are calculated in parallel, and the calculated P*M cluster centers are stored in the DSA storage unit; P is an integer greater than 1; Load each segmented embedding vector and P*M cluster centers in the DSA storage unit into multiple matrix multiplication units in the DSA search engine, and calculate the nearest cluster center of each subvector in each embedding vector in parallel; According to the nearest cluster center of each subvector in each embedded vector, an index table matching the N embedded vectors is constructed and stored in the DSA storage unit, wherein each embedded vector in the index table corresponds to M index values ​​represented by the cluster center.

2. The method according to claim 1, characterized in that: The N sub-vectors in each sub-vector set in the DSA storage unit are loaded into a plurality of matrix multiplication operation units in the DSA search engine, and the P cluster centers corresponding to each sub-vector set are calculated in parallel, including: Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit: Loading N target subvectors from the target subvector set in the DSA storage unit; Based on a preset clustering algorithm, P cluster centers matching the target sub-vector set are calculated through multiple matrix multiplication operations.

3. The method according to claim 1, characterized in that The embedded vectors and P*M cluster centers after each segmentation in the DSA storage unit are loaded into multiple matrix multiplication units in the DSA search engine, and the nearest cluster center of each subvector in each embedded vector is calculated in parallel, including: Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit: Load a target sub-vector in the target embedding vector after segmentation in the DSA storage unit, and load P target cluster centers corresponding to the sub-vector set to which the target sub-vector belongs among the P*M cluster centers; The distance values ​​between the one target sub-vector and the P target cluster centers are calculated respectively by using a distance calculation formula, and the target cluster center with the smallest distance value is determined as the nearest cluster center corresponding to the one target sub-vector.

4. The method according to any one of claims 1 to 3, characterized in that: The embedding vector is a vector representation of external knowledge used by the large language model during model fine-tuning or model inference.

5. An embedding vector retrieval method, characterized in that: Executed by a domain specific architecture DSA, the method comprises: Obtain the embedded vector to be searched, and after the embedded vector to be searched is divided into M search sub-vectors by a division unit in the DSA search engine, each search sub-vector is distributed to a plurality of matrix multiplication operation units in the DSA search engine by a movement unit in the DSA search engine; Load the P*M cluster centers corresponding to the N embedded vectors stored in the DSA storage unit into multiple matrix multiplication units in the DSA search engine, and calculate the distance table corresponding to the embedded vector to be retrieved in parallel, where the distance table contains P*M distance values; Loading the index table stored in the DSA storage unit and constructed by the method according to any one of claims 1 to 4 into a plurality of matrix multiplication operation units, and calculating the distance and value between each embedded vector and the embedded vector to be retrieved in parallel; The N distances and values ​​corresponding to the N embedded vectors are loaded into the topk solution unit in the DSA search engine, and the K embedded vectors with the smallest distances to the embedded vector to be retrieved are calculated as the retrieval results for the embedded vector to be retrieved, where K is an integer greater than or equal to 1.

6. The method according to claim 5, characterized in that The P*M cluster centers corresponding to the N embedded vectors stored in the DSA storage unit are loaded into multiple matrix multiplication operation units in the DSA search engine, and the distance table corresponding to the embedded vector to be retrieved is calculated in parallel, including: Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit: Among the P*M cluster centers stored in the DSA storage unit, load the P target cluster centers that match the target retrieval subvector distributed to this unit; Calculating P distance values ​​between the target retrieval subvector and the P target cluster centers; After obtaining P distance values ​​corresponding to each search sub-vector of the embedded vector to be searched, a distance table consisting of P*M distance values ​​is organized.

7. The method according to claim 5, characterized in that The index table stored in the DSA storage unit is loaded into multiple matrix multiplication units, and the distance and value between each embedded vector and the embedded vector to be retrieved are calculated in parallel, including: Multiple matrix multiplication units in the DSA search engine are triggered in parallel, and the following operations are performed in parallel by each matrix multiplication unit: Loading M target index values ​​corresponding to the target embedding vector from the index table stored in the DSA storage unit; Query the distance table according to the M target index values ​​to obtain M distance values ​​corresponding to the target embedding vector; The M distance values ​​are summed to obtain the distance sum value corresponding to the target embedding vector.

8. The method according to claim 5, characterized in that The N distance sum values ​​corresponding to the N embedding vectors are loaded into the topk solving unit in the DSA search engine to calculate the K embedding vectors with the smallest distance to the embedding vector to be retrieved, including: The topk solving unit selects K distance values ​​with the smallest distances from the N distance sum values ​​based on a preset priority queue algorithm, and obtains K embedding vectors matching the K distance values.

9. A DSA, characterized in that: include: off-chip storage unit, storage controller, and search engine; The search engine includes an on-chip storage unit, an address calculation unit, an on-chip register unit, a plurality of matrix multiplication units, a segmentation unit, a moving unit, a topk solving unit and a cache unit; Among them, the units in the DSA architecture cooperate together to implement the accelerated construction method of the embedded table as described in any one of claims 1-4, or implement the embedded vector retrieval method as described in any one of claims 5-8.

10. An electronic device comprising the DSA according to claim 9.

Images