Vector data processing method and device and electronic equipment

Abstract

The invention relates to a vector data processing method and device and electronic equipment, and the method comprises the steps: carrying out the clustering processing of input vector data through a CPU in a server, and constructing a distance query table according to the clustering result of the input vector data; sending the vector processing task based on the distance lookup table to a persistent PIM kernel of memory computing hardware PIM through a message queue; the persistent PIM kernel comprises a plurality of processing units (PU); for each processing unit (PU), receiving query distribution data of a vector processing task through a heat transmission mechanism, and based on a bus ownership switching mechanism, carrying out vector distance calculation by utilizing vector data stored in a main memory (MRAM) to obtain a distance calculation result; and through the persistent PIM kernel, the distance calculation result after the PU sorting of each processing unit is transmitted back to the CPU for merging processing, and a vector data processing result of the vector processing task is obtained. By adopting the method, the utilization rate of the PU resources and the vector data retrieval efficiency can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to a vector data processing method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] With the development of big data technology, the scale of data will enter the era of YB-level data, of which more than 90% is unstructured data. Approximate Nearest Neighbor Search (ANNS) is a major method for retrieving massive amounts of unstructured data and is widely used in information retrieval, large language models and other fields. However, ANNS has extremely high memory access intensity, with a computation-to-memory access ratio close to 1:1, which is significantly limited by the "memory wall", making it difficult for traditional hardware to meet the performance requirements.

[0003] In related technologies, traditional solutions have two major limitations: For ANNS systems based on CPU / GPU architecture, the CPU is limited by DRAM bandwidth, which cannot match its computing power, while the GPU, although it has high-bandwidth memory, has limited memory capacity, which is difficult to meet the storage requirements of high-dimensional billions of vectors; For PIM-based solutions, its multi-core architecture can alleviate the memory wall problem, but the programming model relies on the batch scheduling paradigm, which has serious efficiency problems, such as long-term PU idle time caused by inter-batch bus arbitration and unbalanced load within batches, resulting in the measured performance being lower than the theoretical value.

[0004] Therefore, traditional ANNS systems suffer from performance bottlenecks and low processing efficiency due to memory walls. Summary of the Invention

[0005] This disclosure provides a vector data processing method, apparatus, electronic device, storage medium, and computer program product to at least solve the problem of low efficiency in vector data processing in related technologies. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a vector data processing method is provided, applied to a server configured with in-memory computing hardware (PIM), comprising:

[0007] The CPU in the server performs clustering processing on the input vector data, and constructs a distance lookup table based on the clustering results of the input vector data; the distance lookup table stores the pre-calculated mapping relationship between vector distances to support the calculation of the distance between the query vector and the data point;

[0008] The vector processing task based on the distance lookup table is sent to the persistent PIM kernel of the memory computing hardware PIM via a message queue; the persistent PIM kernel contains multiple processing units (PUs), each of which is configured with main memory (MRAM) for caching vector data and intermediate calculation results; the vector processing task is to retrieve target vector data similar to the query vector data from the input vector data.

[0009] For each of the aforementioned processing units (PUs), the query distribution data of the vector processing task is received through a hot transfer mechanism. Based on the bus ownership switching mechanism, the vector data stored in the main memory (MRAM) is used to perform vector distance calculation to obtain the distance calculation result. The distance calculation result includes the distance from the query vector data to all data points in the nearest cluster.

[0010] The persistent PIM kernel sends the distance calculation results of each processing unit (PU) after sorting back to the CPU for merging processing, thereby obtaining the vector data processing result of the vector processing task.

[0011] In one possible implementation, receiving the query distribution data of the vector processing task via a hot transfer mechanism includes:

[0012] The control interface link of the processing unit PU is used to construct a control path for communication between the processing unit PU and the CPU.

[0013] Combining the control path and the preset data path, the system receives query and distribution data for the vector processing task; the data path is used to support the CPU to directly read and write the main memory (MRAM) via the DDR bus.

[0014] In one possible implementation, the bus ownership switching mechanism operates as follows:

[0015] When the processing unit (PU) is in the execution computing state, bus ownership is switched from the CPU to the processing unit (PU); the bus ownership transfer message is sent after the CPU completes the copy operation; the bus ownership includes access ownership of the main memory (MRAM);

[0016] When the processing unit PU is in a state of waiting for the CPU to copy data, the ownership of the bus is switched from the processing unit PU to the CPU.

[0017] In one possible implementation, the method further includes:

[0018] During the query distribution process for the processing unit (PU), based on the selective data copying placement method, the target cluster is copied to multiple processing units (PUs) according to the usage popularity of each cluster in the clustering results; the usage popularity of the target cluster meets the preset popularity condition.

[0019] In one possible implementation, the method further includes:

[0020] By monitoring the frequency of cluster access in real time, dynamic adjustments to the data layout are triggered when changes in cluster popularity are detected.

[0021] A copy of the first cluster is created on a low-load processing unit (PU), and a copy of the second cluster is deleted on a high-load processing unit (PU); the first cluster is a cluster with increasing popularity, and the second cluster is a cluster with decreasing popularity.

[0022] In one possible implementation, the method further includes:

[0023] Construct an online request distribution mapping table; the online request distribution mapping table is used to record the mapping relationship between each cluster and all copies in multiple processing units (PUs), each cluster has multiple copies, and the multiple copies are distributed in different processing units (PUs);

[0024] For any cluster of replicas distributed processing units (PUs), based on the number of tasks waiting to be processed by each processing unit (PU), determine the target processing unit (PU) with the lowest current load, and send a query request for any cluster to the target processing unit (PU).

[0025] According to a second aspect of the present disclosure, a vector data processing apparatus is provided, applied to a server configured with memory computing hardware (PIM), comprising:

[0026] The distance lookup table construction unit is configured to perform clustering processing on the input vector data via the CPU in the server, and construct a distance lookup table based on the clustering results of the input vector data; the distance lookup table stores the pre-calculated mapping relationship between vector distances to support the calculation of the distance between the query vector and the data point;

[0027] A vector processing task sending unit is configured to execute a vector processing task based on the distance lookup table, which is sent via a message queue to the persistent PIM kernel of the memory computing hardware PIM. The persistent PIM kernel includes multiple processing units (PUs), each of which is configured with main memory (MRAM) for caching vector data and intermediate calculation results. The vector processing task is to retrieve target vector data similar to the query vector data from the input vector data.

[0028] The distance calculation result obtaining unit is configured to execute for each of the processing units (PUs), receive query distribution data of the vector processing task through a hot transfer mechanism, perform vector distance calculation using the vector data stored in the main memory (MRAM) based on the bus ownership switching mechanism, and obtain the distance calculation result; the distance calculation result includes the distance from the query vector data to all data points in the nearest cluster;

[0029] The distance calculation result feedback unit is configured to send the sorted distance calculation results of each processing unit (PU) back to the CPU for merging processing through the persistent PIM kernel to obtain the vector data processing result of the vector processing task.

[0030] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0031] processor;

[0032] Memory used to store the processor's executable instructions;

[0033] The processor is configured to execute the instructions to implement the vector data processing method as described in any of the preceding claims.

[0034] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the vector data processing method as described in any of the preceding claims.

[0035] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the vector data processing method as described in any of the preceding claims.

[0036] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0037] The disclosed scheme uses the CPU in the server to perform clustering processing on the input vector data. A distance lookup table is constructed based on the clustering results, storing pre-calculated distance mappings between vectors to support distance calculations between query vectors and data points. The vector processing task based on the distance lookup table is then sent via a message queue to the persistent PIM kernel of the in-memory computing hardware (PIM). This persistent PIM kernel contains multiple processing units (PUs), each configured with main memory (MRAM) for caching vector data and intermediate calculation results. The vector processing task is to retrieve target vector data similar to the query vector data from the input vector data. For each PU, the query distribution data for the vector processing task is received via a hot-transfer mechanism. Based on a bus ownership switching mechanism, vector distance calculations are performed using the vector data stored in the main memory (MRAM) to obtain the distance calculation results. These results include the distances from the query vector data to all data points in the closest cluster. Finally, the persistent PIM kernel sends the sorted distance calculation results from each PU back to the CPU for merging processing, resulting in the vector data processing result of the vector processing task. Thus, by coordinating CPU and PIM hardware, and combining a pre-computed distance lookup table with a persistent kernel, computationally intensive tasks can be efficiently offloaded to PIM, effectively improving vector data retrieval efficiency and achieving low-latency, high-throughput vector similarity retrieval. Furthermore, based on message queues, bus switching, and hot transfer mechanisms, the bus arbitration latency and load imbalance issues in traditional batch scheduling can be effectively resolved, improving the utilization rate of PU resources.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0040] Figure 1 This is a schematic diagram illustrating an IVFPQ algorithm according to an exemplary embodiment.

[0041] Figure 2 This is a schematic diagram illustrating the architecture of a UPMEM system and details of its internal PIM units according to an exemplary embodiment.

[0042] Figure 3 This is a schematic diagram illustrating a traditional batch scheduling paradigm according to an exemplary embodiment.

[0043] Figure 4This is a schematic diagram illustrating a comparison of computing performance according to an exemplary embodiment.

[0044] Figure 5 This is a schematic diagram illustrating the PU activity status in a PIM-based ANNS system according to an exemplary embodiment.

[0045] Figure 6 This is a flowchart illustrating a vector data processing method according to an exemplary embodiment.

[0046] Figure 7 This is a schematic diagram of a PIMANN architecture according to an exemplary embodiment.

[0047] Figure 8 This is a schematic diagram illustrating a persistent PIM kernel according to an exemplary embodiment.

[0048] Figure 9 This is a schematic diagram illustrating a single PU scheduling paradigm for PIMANN according to an exemplary embodiment.

[0049] Figure 10 This is a schematic diagram illustrating an example of a PIMANN processing according to an exemplary embodiment.

[0050] Figure 11 This is a schematic diagram illustrating a PU query distribution according to an exemplary embodiment.

[0051] Figure 12 This is a schematic diagram illustrating a throughput comparison according to an exemplary embodiment.

[0052] Figure 13 This is a schematic diagram illustrating a delay comparison according to an exemplary embodiment.

[0053] Figure 14 This is a schematic diagram illustrating a comparison of the number of active PUs over time, according to an exemplary embodiment.

[0054] Figure 15 This is a schematic diagram illustrating PIM utilization according to an exemplary embodiment.

[0055] Figure 16 This is a flowchart illustrating another vector data processing method according to an exemplary embodiment.

[0056] Figure 17 This is a block diagram illustrating a vector data processing apparatus according to an exemplary embodiment.

[0057] Figure 18 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0058] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0059] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure.

[0060] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0061] To facilitate a better understanding of the embodiments of this disclosure by those skilled in the art, the technical terms or some nouns that may be involved in the embodiments of this disclosure are explained as follows:

[0062] ANNS (Approximate Nearest Neighbor Search) is a technique for quickly finding data points in a large dataset that are most similar to a query vector.

[0063] PIM (Processing-in-Memory): is an architecture that integrates computing power into memory, aiming to reduce data movement overhead.

[0064] UPMEM: A commercial PIM hardware that plugs into a general-purpose server via a DIMM interface and contains multiple processing units (PUs) and a large capacity of memory.

[0065] PU (Processing Unit): This is the basic computing unit in UPMEM. Each PU is a RISC-V core that supports multithreading.

[0066] IVFPQ (Inverted File with Product Quantization) is an efficient ANNS algorithm that combines clustering and vector quantization techniques.

[0067] MRAM (Main RAM): is a large storage area in UPMEM, which is accessed by PU through DMA.

[0068] WRAM (Work RAM): A high-speed cache in UPMEM used for fast computation by the PU.

[0069] LUT (Look-Up Table): A pre-calculated distance table used to speed up distance calculation.

[0070] nprobes: The number of clusters to be searched in the IVFPQ algorithm.

[0071] recall@k: A metric for evaluating the quality of the ANNS system, representing the probability that the first k returned results contain the true nearest neighbor.

[0072] QPS (Queries Per Second): A metric that measures system throughput.

[0073] MUX (Multiplexer): Used to switch DDR bus control between the CPU and PU.

[0074] DDR (Double Data Rate) bus: used for high-speed data transfer between the CPU and memory.

[0075] RISC-V: An open-source Reduced Instruction Set Computer (RISC) architecture.

[0076] SIMD (Single Instruction Multiple Data): A parallel computing technique.

[0077] DIMM (Dual In-line Memory Module): A type of physical memory package.

[0078] DMA (Direct Memory Access): Allows the hardware subsystem to read and write memory directly without CPU intervention.

[0079] RAG (Retrieval-Augmented Generation): A technique that combines retrieval systems and large language models.

[0080] kNN graph (k-Nearest Neighbor graph): is a data structure in which each node is connected to its k nearest neighbors.

[0081] For ANNS (Approximate Nearest Neighbor Search), it can be used for traditional information retrieval tasks, such as image-based search and content retrieval. In recent years, as LLMs (Large Language Models) have become a new frontier in artificial intelligence, ANNS has become a core infrastructure for LLMs to carry knowledge. Through RAG (Retrieval Augmentation), it can help LLMs retrieve reliable, accurate, and timely knowledge to alleviate the illusion problem.

[0082] ANNS exhibits extremely high memory access intensity, requiring repeated readings of large numbers of vectors from memory, leading to massive memory accesses. Examples include approximate searches involving billions of data points or performing approximate queries on datasets with tens of billions of entries. However, ANNS also has a very low computation-to-memory ratio. Typical ANNS algorithms rely on maintaining in-memory data structures to quickly identify potential neighbors. ANNS algorithms can be divided into two categories: clustering-based algorithms and graph-based algorithms. Both rely on traversing data points within their data structures (such as clusters or graphs), resulting in a computation-to-memory ratio close to 1:1.

[0083] This disclosure proposes a vector data processing method by building an ANNS system on UPMEM hardware, with a particular focus on clustering-based algorithms well-suited to UPMEM's massively parallel threading capabilities. Specifically, for the IVFPQ (Inverted File and Product Quantization) algorithm, it incorporates product quantization (PQ) techniques to further reduce computational overhead. Figure 1 The IVFPQ algorithm example shown can include an offline phase and an online phase.

[0084] The offline phase is responsible for preparing the dataset for efficient search by building a cluster-based index. This phase includes two steps: Inverted Index File (IVF) and Product Quantization (PQ). 1. The IVF step uses methods such as K-means to divide the entire dataset into C clusters. 2. The PQ step calculates the residuals, such as the difference between each data point x and its corresponding cluster center c, then divides these residual vectors into M sub-vectors and encodes them using a codebook; each encoded point serves as an identifier, corresponding one-to-one with a specific codeword in the codebook. In the actual implementation, the identifier can be encoded as an 8-bit unsigned integer (uint8).

[0085] For the online phase, based on a given query vector q, the online phase can find the k closest vectors to q in four steps: 1. Cluster filtering: Find nprobes closest clusters by calculating the distance between the cluster centers and q. 2. Lookup table (LUT) construction: The LUT is used to store pre-computed data, significantly optimizing the search process by converting computation to in-memory lookups; specifically, each element LUT[j][i] represents the distance between the j-th vector in the codebook Bi and the i-th segment of the residual vector; the distance from q to a given data point x can be calculated by looking up and accumulating the values ​​in the LUT. 3. Distance calculation: Using the LUT, calculate the distance from q to all data points in the filtered clusters. 4. Top-K identification: Sort all calculated distances and select the k closest vectors as approximate nearest neighbors.

[0086] The following is an introduction to the commercial PIM hardware architecture:

[0087] PIM (Memory Processing) is a classic approach to solving the "memory wall" problem, and various PIM architectures have been proposed over a long period. However, prior to the release of UPMEM, these PIM architectures were limited to simulation or FPGA-based verification. UPMEM, as the first commercially available real-world PIM hardware, marks a significant milestone from theoretical concept to practical application. Therefore, this disclosure focuses on a vector data processing method primarily designed for building an ANNS system using UPMEM.

[0088] UPMEM has the following hardware characteristics:

[0089] 1. Multi-core architecture. For example... Figure 2 Figure a shows the architecture of the UPMEM system and details of its internal UPMEMDIMM (PIM units). The thick lines represent the DDR bus, and the dashed lines represent the control interface. The UPMEM connects to the CPU via DIMM slots, similar to standard DRAM. One UPMEMDIMM consists of 64 PIM units, as shown in Figure a. Figure 2 As shown in Figure b, each unit has two main components: a memory library (MRAM) and a general-purpose processing unit (PU) integrated within the library. Each PU is a 32-bit sequential RISC-V core with a 14-stage pipeline, supporting up to 16 hardware threads. By inserting 20 UPMEMDIMMs, a single server can have a total of 2560 PUs, supporting over 40,000 threads and achieving a combined memory bandwidth of 2.5TB / s.

[0090] Due to hardware manufacturing limitations, the current generation of PIM hardware's PU (Power Processing Unit) has the following two problems: 1. The PU only provides a native integer ALU, and floating-point operations are implemented through software, resulting in UPMEM's floating-point operation speed being much lower than its integer operation speed; 2. All PUs follow a shared-nothing architecture, which means that each PU can only access data in its local memory bank. Any communication between PUs must be done through the host CPU.

[0091] 2. Two-level memory hierarchy. Each PIM unit contains two levels of memory: working memory (WRAM) and main memory (MRAM). WRAM, made of SRAM, is 64KB in size and acts as a faster cache. MRAM has a larger storage capacity (64MB), but its access speed is slower than WRAM. The PU cannot directly access MRAM; data must be transferred from MRAM to WRAM via the on-chip DMA engine.

[0092] 3. Bus sharing on MRAM. MRAM receives access from both the CPU and PU. The CPU is used to send / collect task data, while the PU is used for PIM calculations. However, DIMM-PIM architectures that insert PIM devices into DIMM slots (such as UPMEM) inherently do not support simultaneous memory access by the host CPU and PU. Specifically, as... Figure 2 As shown in Figure b, each PIM unit multiplexes the DDR bus for access by both the PU and the host CPU. Because the existing DDR bus protocol requires deterministic latency behavior, simultaneous access to memory by the CPU and PU is not feasible. To address this limitation, the UPMEM hardware employs a multiplexer (MUX) for manual access arbitration. The MUX is a two-state register, with either a CPU or PU side, allowing exclusive access to MRAM for either the CPU or the PU, respectively. The state of the MUX is controlled by the CPU via a control interface.

[0093] Traditional CPU / GPU-based ANNS system solutions primarily rely on CPUs or GPUs for computational acceleration. CPU solutions utilize multi-core parallel computing and SIMD instructions (such as AVX-512) to improve vector retrieval efficiency and depend on large-capacity DRAM (TB-level) storage for index data; while GPU solutions leverage their high parallel computing capabilities (such as CUDA cores) and high-bandwidth memory (HBM, such as the 3TB / s bandwidth of the NVIDIA H100) to accelerate computationally intensive tasks.

[0094] However, due to the memory wall problem, current CPU / GPU-based ANNS systems are increasingly struggling to meet the rapidly growing demands for data volume and performance. While CPUs can provide terabytes of memory capacity to store large numbers of vectors, their limited DRAM bandwidth (approximately 200 GB / s) restricts vector retrieval performance; and work has shown that modern CPUs equipped with SIMD instructions can typically process 5 billion vector operations per second (computational bandwidth of approximately 500 GB / s), far exceeding DRAM bandwidth. On the other hand, although GPUs offer powerful computing capabilities and high-bandwidth memory, their limited memory capacity (even recent high-end GPUs like the H100 only have 100 GB) makes them unable to handle the storage requirements of billions of vectors with thousands of dimensions.

[0095] For traditional PIM-based ANNS system solutions, the PIM programming model adopts a batch scheduling paradigm and follows an accelerator model similar to that of GPUs. Programmers need to prepare the executable program (kernel) and its data in advance, and then start it on the PIM device. The CPU communicates with PIM through a control interface, sending commands to start / synchronize the kernel or query the running kernel status (e.g., ...). Figure 2 (Figure b in the text).

[0096] Currently, the batch scheduling paradigm followed by programs supporting PIM stems from the limitation that the CPU can only interact with PIM's MRAM at the kernel boundary, either before kernel startup or after kernel completion. This limitation arises from the assumption that once the PU is activated, it will retain bus control until the kernel terminates. Therefore, the CPU cannot read or write to PIM's MRAM while the PU is running. Existing batch scheduling paradigms include three steps (such as...). Figure 3 (As shown). First, the CPU prepares a buffer, fills the task set (i.e., a batch) of each PIM unit, and sends the buffer to the MRAM of all PIMs; next, the CPU starts the PIM kernel, prompting the PIM units to start processing their assigned tasks and generating computation results; finally, the CPU collects all the results and moves them into the host memory.

[0097] Regarding the impact of batch scheduling paradigms on ANNS, this study investigates the shortcomings of existing batch scheduling paradigms in ANNS. For example, a prototype ANNS system can be implemented based on the open-source similarity search tool (Faiss), storing clustering data in MRAM and offloading the distance calculation step (the online phase's Â) to the PIM's PU. Figure 4As shown, it can demonstrate the performance on the SPACE-1B dataset, where a represents the throughput of the standard CPUANNS system (Faiss-CPU) and a preliminary PIM system (StrawmenPIM); the dashed line represents the ideal performance of UPMEM; b represents the load of each PU; the load value is normalized through the heaviest PU.

[0098] The study shows that only 18.2% of the theoretical performance was achieved. Figure 4 The computational performance of the Faiss-CPU and PIM prototype systems was demonstrated. For the prototype system, despite having tens of thousands of threads, PIM's throughput was only comparable to a standard CPU; PIM's performance was far from being fully utilized, and its throughput potential could reach more than 5.5 times its current performance if fully utilized. Further analysis revealed two instances of underutilization.

[0099] (1) Insufficient utilization between batches. For example Figure 5 As shown, the number of active PUs in the PIM-based ANNS system varies over time, displaying the number of active PUs in different time periods. The dashed line represents the maximum number of PUs (2560); ideally, the solid line should be as close to the dashed line as possible. Between two batches, there is a period exceeding 1.21 seconds during which the number of active PUs is zero. This period accounts for 65% of the duration of a single batch, during which the CPU is copying the results of the previous batch from MRAM while simultaneously transferring the input of the next batch to MRAM. Since the CPU and PUs cannot access the shared bus simultaneously, no PUs can be activated during this period.

[0100] (2) Insufficient utilization within a batch. In batch scheduling, a batch of queries is allocated to the PU. For example... Figure 4 As shown, b represents the number of tasks assigned to each PU. Uneven load between PUs may cause some PUs to be idle while others are overloaded, thus delaying the completion of the entire batch.

[0101] To address the following issues faced by traditional CPU / GPU-based ANNS systems and traditional PIM batch scheduling paradigm ANNS systems: 1. Performance bottlenecks caused by the memory wall: CPU solutions are limited by DRAM bandwidth (200GB / s), resulting in severe idle computing resources (SIMD computing bandwidth reaches 500GB / s, but data supply is insufficient); GPU solutions, while possessing high bandwidth (e.g., H100's 3TB / s), have limited video memory capacity (≤100GB), unable to support the storage requirements of billions of high-dimensional vectors. 2. Inefficiency of traditional PIM batch modulation, such as insufficient inter-batch utilization, where the CPU and PU alternately occupy the bus, resulting in 65% of the time being idle (data transfer time accounts for 65% of the batch cycle), and uneven load within batches, where uneven task allocation causes some PUs to be overloaded and others idle, resulting in an overall throughput of only 18.2% of the theoretical value. This disclosure proposes a vector retrieval scheme for ANNS systems built based on UPMEM hardware, which is described in detail below.

[0102] Figure 6 This is a flowchart illustrating a vector data processing method according to an exemplary embodiment. This method can be applied to a server configured with Partition In-Memory (PIM) computing hardware. In this embodiment, the method includes the following steps:

[0103] In step S610, the CPU in the server performs clustering processing on the input vector data, and a distance lookup table is constructed based on the clustering results of the input vector data.

[0104] The distance lookup table stores pre-calculated mappings of distances between vectors to support distance calculations between query vectors and data points.

[0105] In practical applications, such as Figure 7 The PIMANN architecture shown offloads the most parallelizable distance computation step in the entire IVFPQ process to the UPMEM, while other steps are still executed by the CPU. Each PU stores multiple IVF clusters and performs the corresponding distance computations. On the CPU side, the CPU can perform cluster selection on the input vector data and build a distance lookup table (LUT). The LUT can be used to store pre-computed data and supports converting computation into in-memory lookups for searching.

[0106] In step S620, the vector processing task based on the distance lookup table is sent to the persistent PIM kernel of the memory computing hardware PIM via a message queue.

[0107] The persistent PIM kernel contains multiple processing units (PUs), each of which is configured with main memory (MRAM) for caching vector data and intermediate calculation results.

[0108] As an example, a vector processing task can be a task that retrieves target vector data that is similar to query vector data from input vector data.

[0109] Specifically, such as Figure 7 As shown, to alleviate insufficient utilization between batches, a persistent PIM kernel can be introduced through PIMAN. Starting a persistent kernel can support continuous query processing, thereby eliminating idle states between batches. Figure 7 The workflow shown in PIMANN for performing ANNS queries can send vector processing tasks to the persistent PIM kernel on the PIM side via a message queue through a single PU query distribution module.

[0110] In step S630, for each processing unit (PU), query and distribution data of vector processing tasks are received through a hot transfer mechanism. Based on the bus ownership switching mechanism, vector distance is calculated using the vector data stored in main memory (MRAM) to obtain the distance calculation result.

[0111] The distance calculation result can include the distance from the query vector data to all data points in the nearest cluster.

[0112] In the implementation, each PU can independently and continuously receive queries from the CPU. However, due to bus sharing, the native UPMEM driver does not support data copying during PIM runtime. To address this issue, a hot-transfer mechanism can be implemented by modifying the control interface to fine-grained bus arbitration to transfer data between the CPU and PU during PIM runtime. With bus arbitration support, each PU can dynamically switch between two states: Running (performing computation) or WaitCPUCopy (waiting for the CPU to copy data), corresponding to ownership of the shared bus being held by the PU or the CPU, respectively. To alleviate insufficient utilization within a batch, a query distribution method can be used, with the help of bus arbitration, to distribute queries to each PU based on its real-time load. For PUs following a shared-nothing architecture, where different PUs cannot access each other's MRAM (main memory), an efficient data placement scheme can be designed, pre-allocating clusters for each PU.

[0113] For example, each PU in the PIM kernel can receive task data transmitted by the CPU in real time (i.e., querying and distributing data) through a hot transfer mechanism. Then, it can dynamically switch between the Running (performing calculations) and WaitCPUCopy (waiting for the CPU to copy data) states based on the bus ownership switching mechanism. It can use the vector data stored in MRAM to perform distance calculation and Top-K sorting processing, that is, obtain the distance calculation result through vector distance calculation.

[0114] In one example, such as Figure 8 The diagram illustrates the design of a persistent PIM kernel. Each PU can continuously pull tasks from the message queue and execute computations; task input / output data is transferred via a hot transfer mechanism, involving multiple bus ownership switches. Unlike traditional batch scheduling paradigms, PIMANN starts only one persistent PIM kernel for all ANNS queries during system initialization. Within this kernel, the bus for each PU can be dynamically arbitrated at runtime, allowing each PU to independently perform computations or data replication. Thus, the persistent PIM kernel design eliminates the idle replication time introduced by batch scheduling.

[0115] For example, the workflow of each PU can continuously cycle through the following steps:

[0116] 1. Dequeue an ANNS request from the message queue;

[0117] 2. Switch to WaitCPUCopy mode;

[0118] 3. Wait for the CPU to copy the task data (such as LUT) to MRAM;

[0119] 4. Switch to Running mode and perform calculations;

[0120] 5. Switch to WaitCPUCopy mode;

[0121] 6. Wait for the CPU to copy the calculation results.

[0122] In the above steps, since steps 1, 3, and 6 require data transfer from the CPU while the PU is running, and steps 2, 4, and 5 require switching the state of the PU (such as switching ownership of the shared bus), a hot transfer mechanism and a bus ownership switching scheme can be designed respectively.

[0123] In step S640, the distance calculation results after sorting each processing unit (PU) are sent back to the CPU for merging processing through the persistent PIM kernel to obtain the vector data processing results of the vector processing task.

[0124] After obtaining the distance calculation results, such as Figure 7 As shown, the PIM kernel can send the calculation results of each PU back to the CPU via a message queue. The CPU can then merge and integrate the sorted distance calculation results from multiple PUs, and finally output the vector data processing results based on the requirements of the vector processing task, thus completing the vector data processing flow based on PIM-CPU collaboration. Optionally, the vector data processing results may include a Topk set of similar vectors, clustering classification results, etc.

[0125] Compared to traditional methods, traditional PIM systems use a batch scheduling mode where the CPU and PU alternately operate on MRAM, which leads to insufficient utilization between and within batches. The root cause of this problem is the assumption that the PU holds control of the shared bus throughout the entire execution of the PIM kernel (from startup to completion).

[0126] This embodiment breaks with this assumption by designing a fine-grained scheduling and arbitration system based on each PU. Each PU in UPMEM has an additional control interface (originally used for control commands, which can be modified to send custom control messages) to finely coordinate PU and CPU access. Based on this, this embodiment proposes a per-PU scheduling paradigm (such as...). Figure 9 ), combined Figure 3 and Figure 9 This allows for a comparison between the traditional batch scheduling paradigm and the PIMANN per-PU scheduling paradigm. For example... Figure 9 As shown, for the single-PU-based scheduling paradigm in PIMAN, by manipulating the MRAM ownership of each PU unit, it can be determined whether the read and write operations are performed by the PU or the CPU.

[0127] Each PU scheduling paradigm has the following advantages: it can eliminate idle states between two batches (e.g., ... Figure 5 This effectively solves the problem of insufficient utilization between batches; each PU can be in one of two states, performing computation or waiting for CPU to copy data. It also eliminates the need for group scheduling, dynamically distributing queries based on the real-time load of the PUs, such as... Figure 7 In the case of PU0 overload, the query can be distributed to a replica on PU1 based on the query scheduling of PU; thus achieving good load balancing and avoiding insufficient utilization within a batch.

[0128] To achieve per-PU scheduling, the following issues need to be addressed: 1. The native driver does not allow reading or copying data from the PIM during PIM runtime; 2. The read / write bandwidth between the CPU and PU's MRAM is limited, and switching PU states may cause the CPU to experience prolonged stagnation; 3. Although per-PU scheduling can dynamically send queries based on the PU's current load, PUs do not share memory space. This embodiment's technical solution designs an efficient data placement scheme that can pre-assign clusters to PUs.

[0129] Regarding the overall architecture, such as Figure 10 As shown, the PIM architecture in this embodiment allows computation to be located close to storage (MRAM), thereby reducing data transfer time. At the same time, combined with clustering optimization and dynamic task scheduling on the CPU side, it can significantly improve vector retrieval efficiency to adapt to YB-level unstructured data processing scenarios.

[0130] In the above vector data processing method, the CPU in the server performs clustering processing on the input vector data, constructs a distance lookup table based on the clustering results, and then sends the vector processing task based on the distance lookup table to the persistent PIM kernel of the memory computing hardware (PIM) via a message queue. For each processing unit (PU), the query distribution data of the vector processing task is received through a hot transfer mechanism. Based on the bus ownership switching mechanism, vector distance calculation is performed using the vector data stored in main memory (MRAM) to obtain the distance calculation result. This distance calculation result includes the distance from the query vector data to all data points in the nearest cluster. Then, the persistent PIM kernel sends the sorted distance calculation results of each processing unit (PU) back to the CPU for merging processing to obtain the vector data processing result of the vector processing task. Thus, by coordinating CPU and PIM hardware, and combining a pre-computed distance lookup table with a persistent kernel, computationally intensive tasks can be efficiently offloaded to PIM, effectively improving vector data retrieval efficiency and achieving low-latency, high-throughput vector similarity retrieval. Furthermore, based on message queues, bus switching, and hot transfer mechanisms, the bus arbitration latency and load imbalance issues in traditional batch scheduling can be effectively resolved, improving the utilization rate of PU resources.

[0131] In an exemplary embodiment, receiving query distribution data of a vector processing task via a hot transfer mechanism includes: using a control interface link of a processing unit (PU) to construct a control path for communication between the processing unit (PU) and the CPU; combining the control path and a preset data path to receive the query distribution data of the vector processing task; the data path is used to support the CPU to directly read and write main memory (MRAM) via the DDR bus.

[0132] In practical applications, a hot-transfer mechanism can be used to enable data transfer between the CPU and PU during PU runtime. This hot-transfer mechanism can consist of two parts: a separate control path and a separate data path. The control path can use a custom message queue and be isolated from the DDR bus via the PU's control interface link. The data path allows the CPU to directly read and write MRAM via the DDR bus (but requires ownership of the bus).

[0133] For the control path, it can be a message queue implemented based on the control interface. To achieve fine-grained per-PU scheduling, the most basic function is to enable CPU-PU communication during PIM runtime. Analysis of the UPMEM driver reveals that CPU read / write access to WRAM is performed through an additional, undocumented control interface link, completely bypassing the DDR bus. Based on this hardware characteristic, a message queue can be directly implemented on the WRAM for communication between the PU and CPU.

[0134] In one alternative embodiment, the message queue has two limitations that need to be considered when designing the system: 1. The message queue has a limited capacity (e.g., tens of bytes), while the typical size of a common data structure LUT in ANNS queries is 32KB; therefore, only necessary metadata (e.g., query ID) or small control messages (e.g., ownership transfer) can be stored in this queue, while other large messages (e.g., LUTs) are written directly to MRAM. 2. Since PIM cannot interrupt the CPU, PIMANN can maintain a dedicated CPU thread to poll the message queues of all PUs to obtain PIM notifications.

[0135] For the data path, it can expose MRAM to the CPU. The native UPMEM driver does not support exposing MRAM to the CPU for direct read and write operations during PIM runtime. Analysis of the driver can lead to the following modifications: 1. Expose MRAM to the CPU during PIM runtime (e.g., enabling the CPU to directly read and write MRAM); 2. Construct a mapping table to map all variable symbols to their corresponding MRAM addresses (e.g., determining the location the CPU should read or write).

[0136] For modification 1, all MRAMs are treated as multiple device files in the operating system and mapped to virtual memory space. The base address of the mmap file can be obtained by injecting a recording function in the PIM boot path. Considering that UPMEM uses memory-level parallelism, all write operations to MRAM should transpose the data to ensure that the data can be correctly transferred to the same chip.

[0137] For Modification 2, the main challenge is the lack of a mapping from variable symbols to mmap file offsets. The solution is as follows: Before starting PIM, the corresponding variable can be accessed once using the standard MRAM copy interface (dpu_copy_to), the corresponding address can be recorded, and then cached in the user program.

[0138] Optionally, the bandwidth for direct CPU writes to MRAM during runtime is relatively low (0.41 GB / s) because, compared to the batch scheduling paradigm, which can easily utilize bank-level parallelism (8 PUs), fine-grained per-PU scheduling means that only one PU's MRAM can be copied at a time. However, from the perspective of a single PU, its data copy latency remains unchanged, so fine-grained scheduling does not increase the latency of the ANNS task.

[0139] The technical solution of this embodiment adopts the control interface link of the processing unit (PU). The control interface link is used to construct a control path for communication between the processing unit (PU) and the CPU. Then, by combining the control path and the preset data path, the query and distribution data of the vector processing task is received. This can realize an efficient communication architecture between the CPU and the PU, which helps to improve the parallel efficiency and resource utilization of PIM hardware.

[0140] In an exemplary embodiment, the bus ownership switching mechanism operates as follows: when the processing unit PU is in the state of performing computation, bus ownership is switched from the CPU to the processing unit PU; the bus ownership transfer message is sent after the CPU completes the copy operation; bus ownership includes access ownership of main memory MRAM; when the processing unit PU is in the state of waiting for the CPU to copy data, bus ownership is switched from the processing unit PU to the CPU.

[0141] In the specific implementation, based on the bus ownership switching mechanism, the state of each PU can be switched safely and efficiently in PIMANN, such as the Running (performing computation) state or the WaitCPUCopy (waiting for the CPU to copy data) state. Since bus ownership is controlled by the MUX register, the MUX register mapping of each PU unit can be exposed to user space by referring to the UPMEM driver code. Each MUX has two states: CPU side and PU side. If memory access is inconsistent with the MUX state, it may lead to abnormal access by the CPU or PU. To ensure state consistency in PIMANN, the following discusses the two PU state switching scenarios: 1. WaitCPUCopy → Running (MRAM access ownership switches from CPU to PU); 2. Running → WaitCPUCopy (reverse).

[0142] In one example, for WaitCPUCopy→Running, in PIMANN, after the CPU completes the copy operation, it can send an ownership transfer message to the queue. The PU then only needs to poll the message queue to receive the message, and can then safely access MRAM and perform computations.

[0143] For the Running → WaitCPUCopy state, since the PU itself cannot switch its MUX, it can send a message to the CPU via a queue. After receiving the message, the CPU can switch its MUX and send an acknowledgment message to the PU. At this point, the PU can safely enter the WaitCPUCopy state.

[0144] To reduce software stack overhead, PIMANN caches the MUX state of each PU in user space. This cache can be directly accessed via an interface that checks the MUX state. When PU bus ownership changes, the actual MUX state and its cached copy are updated simultaneously. Since only the CPU can modify the MUX state, cache inconsistency issues are resolved.

[0145] In another example, regarding paired ownership switching, during implementation, when the CPU directly accesses the MRAM of a PU, it can sometimes cause tasks on adjacent PUs to malfunction. Since two adjacent PUs may share the same physical MUX register, this is likely due to the unique hardware circuit design of UPMEM. For example, if the MUXs of two adjacent PUs share a single control interface, then the smallest granularity of PIMANN switching is determined to be two adjacent PUs. This means that even if the current query only involves one PU, ownership of both adjacent PUs must be switched simultaneously, resulting in a waste of resources for the other adjacent PU.

[0146] To address this limitation and reduce resource waste, a pairwise clustering and sharding optimization scheme can be adopted to ensure that adjacent PUs are always invoked simultaneously. Specifically, instead of assigning a single IVF cluster to a single PU, each cluster is divided into two smaller cluster shards, and adjacent PUs are assigned to different shards within the same cluster. This way, when one PU in a pair is activated, its adjacent PUs are also activated synchronously, thereby maximizing hardware resource utilization.

[0147] Addressing hidden switching latency. Single-PU bus ownership switching can introduce significant latency when the required MUX ownership is not immediately available. This is primarily due to two reasons: First, the PU's computing power is relatively weak, with a clock frequency only on the order of O(100) MHz, resulting in potentially long task execution times for a single PU; second, CPU-PIM communication latency is high. Even if the PU releases ownership, the CPU cannot immediately acquire it, but instead needs to poll the message queue on WRAM through a bandwidth-limited interface. For example, polling the message queue for one rank (64 PUs) takes 0.9 milliseconds. Therefore, simply blocking will lead to inefficient CPU threads.

[0148] To mitigate this issue, a coroutine-based optimization scheme can be employed, allowing the CPU to switch execution contexts to handle other tasks while waiting for ownership to switch, thereby hiding the switching latency. Specifically, coroutine scheduling can follow these priority principles: 1. Immediate availability principle: PU-related coroutines whose bus ownership has switched to the CPU side are executed first; 2. Predictive scheduling principle: When the MUX state is still on the PU side, the scheduler estimates the state switching timing based on the fixed execution time of the PU task and assigns higher priority to coroutines that are about to become ready. This estimation leverages the deterministic characteristics of the UPMEM sequential RISC core (which lacks caching or speculative execution mechanisms).

[0149] The technical solution of this embodiment, by switching bus ownership from the CPU to the processing unit PU when the processing unit PU is in the state of performing calculations, and switching bus ownership from the processing unit PU to the CPU when the processing unit PU is in the state of waiting for the CPU to copy data, can avoid access conflicts based on dynamic switching of bus ownership and effectively reduce PU idle time to improve computing efficiency.

[0150] In one exemplary embodiment, the method further includes: during the query distribution process for the processing unit PU, based on the selective data copying placement method, the target cluster is copied to multiple processing units PU according to the usage popularity of each cluster in the clustering results; the usage popularity of the target cluster meets a preset popularity condition.

[0151] Specifically, in the per-PU query distribution process, the persistent PIM kernel design can distribute queries to each PU based on the real-time load of the PU, thereby solving the problem of insufficient utilization within a batch. The per-PU query distribution design is as follows: Figure 11 As shown. However, since each PU follows a shared-nothing architecture, different PUs cannot access each other's MRAM. Therefore, a suitable data (i.e., clustering) placement strategy needs to be designed. If clusters are randomly assigned to PUs, significant load imbalance often occurs due to changes in cluster popularity. To address this, PIMANN employs a selective replication data placement scheme, which selectively replicates popular (high-popularity) clusters to multiple PUs.

[0152] For example, selective copying of data placement may include the following steps:

[0153] 1. Replicating High-Popularity Clusters. Clusters can be replicated based on their popularity, which is quantified as the product of cluster size (si) and access frequency (fi). The number of replicas of a cluster can be determined by dividing its popularity (pi = si * fi) by the average PU load. For a high-popularity cluster i, its replica count i = pi / pavg, where pavg = ∑pi, and n represents the number of clusters.

[0154] 2. Slicing Clustering to Reduce Memory Fragmentation. Since the size of the MRAM is not necessarily a precise multiple of the cluster size, the above strategies may lead to significant memory fragmentation. To address this issue, clusters can be sliced ​​into smaller, uniformly sized segments. Specifically, the MRAM can be divided into fixed-length slots, each accommodating one segment. This simplifies data allocation and minimizes memory fragmentation.

[0155] 3. Cluster Segment Placement. Cluster segments are placed to ensure balanced utilization among PUs. Due to the constraints of paired bus ownership switching, adjacent PUs can be grouped, and segments can be evenly distributed among PU pairs. A heat cap can also be defined for each PU pair; this heat cap is a hyperparameter used to balance the time required for generation and placement with the degree of load balancing. Within this cap, all cluster segments are allocated to maintain load balance.

[0156] In an alternative embodiment, for selective data placement processing, in addition to using a greedy strategy to allocate clustered replicas, the problem can also be modeled as a packing problem to more rigorously optimize PU resource utilization. Specifically, the following implementation method can be adopted:

[0157] 1. Problem Modeling: Each PU is treated as a "box," the capacity of which is determined by the remaining MRAM space. The attributes of each cluster segment include storage space requirements and computational load (positively correlated with access frequency). The objective is to minimize the load on each PU while ensuring that the storage of each PU does not exceed its limits.

[0158] 2. Solution Method: Heuristic algorithms (such as FFD, First-Fit Decreasing Algorithm) or integer linear programming are used to find approximate optimal solutions. By periodically (e.g., after processing 1000 queries) recalculating the optimal allocation, the system can dynamically adapt to changes in popularity.

[0159] The technical solution of this embodiment, by selectively copying data placement during the query distribution process for processing units (PUs), copies the target cluster to multiple processing units (PUs) according to the usage popularity of each cluster in the clustering results. This enables the query task to be processed nearby and accelerated in parallel, improving the efficiency of vector data retrieval and balancing the PU load.

[0160] In one exemplary embodiment, the method further includes: triggering dynamic data layout adjustment processing when the cluster popularity is determined to change by real-time monitoring of cluster access frequency; creating a copy of the first cluster on a low-load processing unit PU and deleting a copy of the second cluster on a high-load processing unit PU; wherein the first cluster is a cluster with increasing popularity and the second cluster is a cluster with decreasing popularity.

[0161] In one example, cluster popularity may change over time, requiring real-time adjustments to the data layout. A persistent kernel-based scheduling design based on PUs can dynamically adjust the data layout by detecting popularity changes, enabling the reallocation of clusters without disabling PIM.

[0162] For detecting changes in popularity, a lightweight monitoring mechanism can be used to track the access frequency of clusters in real time. Specifically, a frequency counter can be assigned to each cluster, and the counter increments when a query involves that cluster. Significant changes in cluster popularity can be detected through predefined time windows and thresholds. For example, within a preset time window (such as the past 1000 queries), if the access frequency of a certain cluster changes by more than a preset threshold (such as doubling) compared to the previous time window, regardless of whether it increases or decreases, it can be considered that the popularity of that cluster has changed significantly.

[0163] In another example, the adjustment strategy based on the update popularity can be: 1. For clusters with increasing popularity (i.e., the first cluster), additional replicas can be created on low-utilization PUs based on the latest popularity, and queries can be dynamically redirected to the new replicas; 2. For clusters with decreasing popularity (i.e., the second cluster), redundant replicas can be removed to free up MRAM space, and replicas on high-load PUs can be deleted first. The freed memory space can be reallocated to clusters with increasing popularity.

[0164] The technical solution of this embodiment monitors the cluster access frequency in real time and triggers dynamic adjustment of data layout when the cluster popularity changes. Then, it creates a copy of the first cluster on the low-load processing unit PU and deletes a copy of the second cluster on the high-load processing unit PU. Based on the dynamic adjustment of the distribution of cluster copies, it can improve parallel processing efficiency, realize load balancing and storage utilization optimization, and improve the overall throughput of PIM.

[0165] In one exemplary embodiment, the method further includes: constructing an online request distribution mapping table; the online request distribution mapping table is used to record the mapping relationship between each cluster and all replicas in multiple processing units (PUs), each cluster has multiple replicas, and the multiple replicas are distributed in different processing units (PUs); for the processing units (PUs) where the replicas of any cluster are distributed, the target processing unit (PU) with the lowest current load is determined according to the number of tasks waiting to be processed by each processing unit (PU), and a query request for any cluster is sent to the target processing unit (PU).

[0166] In practical applications, online request distribution can be employed. A mapping table is maintained on the host side, recording the mapping from cluster IDs to PUIDs (e.g., all replicas). The distributor always selects the lightest-loaded PU (i.e., the target processing unit) to send requests. The load of a PU can be represented by the message queue depth, such as the number of tasks waiting to be processed. This dynamic load balancing prioritizes distributing query requests to idle PUs, improving PIM hardware utilization. Furthermore, the redundant distribution of multiple replicas avoids single-PU overload or bus contention, further reducing task latency.

[0167] The technical solution of this embodiment constructs an online request distribution mapping table. For the processing units (PUs) distributed among the replicas of any cluster, the target processing unit with the lowest current load is determined based on the number of tasks waiting to be processed by each processing unit (PU). A query request for any cluster is sent to the target processing unit (PU). This can effectively improve the parallel processing capability and resource utilization of vector data processing and reduce the access latency of hot PUs.

[0168] The following combination Figure 12-15 Overall, the beneficial effects of the technical solutions provided by the embodiments of this disclosure are explained as follows:

[0169] 1. By comparing PIMANN with the following ANNS systems:

[0170] (1) Faiss-CPU: A popular approximate nearest neighbor search library.

[0171] (2) PIMANN-Batch: The basic version of PIMANN, which adopts the batch scheduling paradigm. PIMANN-Batch also enables the same load balancing strategy as PIMANN (including selective replication).

[0172] The datasets were evaluated using the following two large-scale datasets:

[0173] SIFT-1B: Contains 1 billion 128-dimensional vectors, encoded as 32 segments.

[0174] SPACE-1B: Contains 1 billion 100-dimensional vectors, encoded as 20-dimensional vectors.

[0175] Metrics settings: Compare throughput (QPS), latency, and tail latency of different solutions under the same recall@10 condition. recall@10 refers to the recall rate of the first 10 results returned by the ANNS system.

[0176] The following default experimental configurations are used: selective replication uses all MRAM memory by default; the number of clusters in IVF is set to 4096; the number of coroutines is 4; the slice size is 100K; the default dataset is SPACE-1B; and the default recall@10 is 0.9.

[0177] 2. Regarding the relationship between performance and recall rate

[0178] Through evaluation of throughput and latency of PIMANN, Faiss-CPU, and PIMANN-Batch, PIMANN significantly outperformed the other two in all cases.

[0179] (1) Overall throughput. Figure 12 The throughput of PIMANN, Faiss-CPU, and PIMANN-Batch on SPACE-1B and SIFT-1B was demonstrated. The following results can be drawn: 1. PIMANN-Batch's throughput is 2.4-3.7 times higher than Faiss-CPU; this is because batch scheduling utilizes the high memory bandwidth of UPMEM. 2. Compared to PIMANN-Batch, introducing per-PU scheduling (PIMANN) can further improve throughput by 2.4-2.9 times; this is because PIMANN eliminates the underutilization between batches caused by batch group scheduling and alleviates the underutilization caused by uneven PU load within a batch. Therefore, PIMANN's QPS is 5.9-10.4 times higher than Faiss-CPU. 3. Under low recall (≤0.88), the QPS of the three systems on SPACE-1B is higher than that on SIFT-1B, while the opposite is true under high recall. This is because under low recall, SPACE-1B requires fewer clusters to be queried compared to SIFT-1B, while the opposite is true under high recall.

[0180] (2) End-to-end delay. Figure 13 The presentation showcases the latency of Faiss-CPU, PIMANN-Batch, and PI-MANN at different recall rates. PIMANN significantly outperforms both CPU and PIMANN-Batch in performance. Compared to Faiss-CPU, although PIMANN-Batch features highly parallel UPMEM and achieves a 2.4-2.9x increase in throughput, its latency is approximately 7.1 to 10.5 times higher, primarily due to additional inter-batch blocking latency. In contrast, PIMANN eliminates this additional latency by enabling fine-grained bus sharing, reducing latency by 31-40%. Independent per-PU scheduling allows tasks to be scheduled independently, thereby reducing end-to-end per-query latency.

[0181] 3. Regarding PIM utilization rate

[0182] By studying the PIM utilization rate of different systems Figure 14 This shows the number of active computational units (PUs) changing over time. Active PUs refer to PUs currently performing computational tasks. The dataset used here is SPACE-1B, with a recall of 0.9; similar results are obtained for other datasets.

[0183] like Figure 14 As shown, the number of active PUs in PIMAN (solid line) remains relatively stable over time, with a utilization rate of approximately 80%. This stability is due to PIMAN overcoming the limitations of batch scheduling. When a CPU receives a query task and completes the corresponding stage, it can immediately send the task to a PU. After completing the task, the PU can immediately return the result to the CPU and wait for new tasks. The PU remains active from receiving the task until it is completed. In contrast, the number of active PUs in a PIMAN-Batch exhibits periodic fluctuations (dashed line). At the beginning of each batch, the number of active PUs peaks when all PUs are initially active. Subsequently, during the data replication phase between batches, all PUs relinquish bus ownership and become inactive. Furthermore, within a batch, due to uneven task loads among PUs, some PUs complete their tasks early and enter a waiting state, further reducing the number of active PUs.

[0184] By integrating the change in the number of active PUs over time, and based on this, calculating the PIM utilization rate of the two systems under different datasets and recall conditions, such as... Figure 15 As shown, across all datasets and all recall rates, PIMANN's PIM utilization can reach 65-83%, far exceeding the batch scheduling paradigm. (20%), which demonstrates the effectiveness and efficiency of the PIMANN design; the persistent PIM kernel technology eliminates idle states between batches. On the other hand, the per-PU distribution technology ensures that the load on each PU is fully balanced.

[0185] Figure 16 This is a flowchart illustrating another vector data processing method according to an exemplary embodiment. In this embodiment, the method includes the following steps.

[0186] In step S1610, the CPU in the server performs clustering processing on the input vector data, and constructs a distance lookup table based on the clustering results. In step S1620, the vector processing task based on the distance lookup table is sent to the persistent PIM kernel of the memory computing hardware PIM via a message queue. In step S1630, the control interface link of the processing unit (PU) is used to construct a control path for communication between the processing unit (PU) and the CPU. In step S1640, the query distribution data of the vector processing task is received by combining the control path and the preset data path. In step S1650, during the query distribution process for the processing unit (PU), based on the selective data replication placement method, the target cluster is copied to multiple processing units (PUs) according to the usage frequency of each cluster in the clustering results. In step S1660, based on the bus ownership switching mechanism, the vector distance is calculated using the vector data stored in the main memory (MRAM) to obtain the distance calculation result. In step S1670, the distance calculation results after sorting each processing unit (PU) are sent back to the CPU for merging processing through the persistent PIM kernel to obtain the vector data processing results of the vector processing task.

[0187] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a vector data processing method described above, and will not be repeated here.

[0188] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0189] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0190] Based on the same inventive concept, this disclosure also provides a vector data processing apparatus for implementing the vector data processing method described above.

[0191] Figure 17This is a block diagram illustrating a vector data processing apparatus according to an exemplary embodiment. The apparatus is applied to a server configured with in-memory computing hardware (PIM), and includes:

[0192] The distance lookup table construction unit 1710 is configured to perform clustering processing on the input vector data by the CPU in the server, and construct a distance lookup table based on the clustering results of the input vector data; the distance lookup table stores the pre-calculated mapping relationship between vector distances to support the calculation of the distance between the query vector and the data point;

[0193] The vector processing task sending unit 1720 is configured to execute a vector processing task based on the distance lookup table, which is sent via a message queue to the persistent PIM kernel of the memory computing hardware PIM. The persistent PIM kernel includes multiple processing units (PUs), each of which is configured with main memory (MRAM) for caching vector data and intermediate calculation results. The vector processing task is to retrieve target vector data similar to the query vector data from the input vector data.

[0194] The distance calculation result obtaining unit 1730 is configured to execute for each of the processing units PU, receive the query distribution data of the vector processing task through a hot transfer mechanism, and perform vector distance calculation using the vector data stored in the main memory MRAM based on the bus ownership switching mechanism to obtain the distance calculation result; the distance calculation result includes the distance from the query vector data to all data points in the nearest cluster;

[0195] The distance calculation result feedback unit 1740 is configured to send the sorted distance calculation results of each processing unit PU back to the CPU for merging processing through the persistent PIM kernel to obtain the vector data processing result of the vector processing task.

[0196] In one possible implementation, the distance calculation result obtaining unit 1730 is specifically configured to execute a control interface link using the processing unit PU, and construct a control path for communication between the processing unit PU and the CPU through the control interface link; in combination with the control path and a preset data path, it receives query distribution data of the vector processing task; the data path is used to support the CPU to directly read and write the main memory MRAM through the DDR bus.

[0197] In one possible implementation, the bus ownership switching mechanism operates as follows: when the processing unit PU is in the execution computing state, bus ownership is switched from the CPU to the processing unit PU; the bus ownership transfer message is sent after the CPU completes the copy operation; the bus ownership includes access ownership of the main memory MRAM; when the processing unit PU is in the waiting state for the CPU to copy data, bus ownership is switched from the processing unit PU to the CPU.

[0198] In one possible implementation, the vector data processing apparatus further includes:

[0199] The selective data placement unit is specifically configured to, during the query distribution process for the processing unit (PU), copy the target cluster to multiple processing units (PUs) based on the selective data placement method and according to the usage popularity of each cluster in the clustering results; the usage popularity of the target cluster meets a preset popularity condition.

[0200] In one possible implementation, the vector data processing apparatus further includes:

[0201] The cluster access frequency monitoring unit is specifically configured to perform dynamic data layout adjustment processing by monitoring the cluster access frequency in real time and determining that the cluster popularity has changed.

[0202] The clustering copy adjustment unit is specifically configured to create a copy of the first cluster on a low-load processing unit PU and delete a copy of the second cluster on a high-load processing unit PU; the first cluster is a cluster with increasing popularity and the second cluster is a cluster with decreasing popularity.

[0203] In one possible implementation, the vector data processing apparatus further includes:

[0204] The mapping table construction unit is specifically configured to construct an online request distribution mapping table; the online request distribution mapping table is used to record the mapping relationship between each cluster and all copies in multiple processing units (PUs), each cluster has multiple copies, and the multiple copies are distributed in different processing units (PUs);

[0205] The online request distribution unit is specifically configured to execute the processing unit (PU) distributed for the replicas of any cluster, determine the target processing unit (PU) with the lowest current load based on the number of tasks waiting to be processed by each processing unit (PU), and send the query request for any cluster to the target processing unit (PU).

[0206] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0207] Each module in the aforementioned vector data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0208] Figure 18 This is a block diagram illustrating an electronic device 1800 for implementing a vector data processing method according to an exemplary embodiment. For example, the electronic device 1800 may be a server. (Refer to...) Figure 18 The electronic device 1800 includes a processing component 1820, which further includes one or more processors, and memory resources represented by memory 1822 for storing instructions, such as application programs, that can be executed by the processing component 1820. The application programs stored in memory 1822 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1820 is configured to execute instructions to perform the methods described above.

[0209] Electronic device 1800 may further include: a power supply component 1824 configured to perform power management of electronic device 1800, a wired or wireless network interface 1826 configured to connect electronic device 1800 to a network, and an input / output (I / O) interface 1828. Electronic device 1800 may operate on an operating system stored in memory 1822, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0210] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1822 including instructions, which can be executed by a processor of an electronic device 1800 to perform the above-described method. The storage medium may be a computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0211] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor of an electronic device 1800 to perform the above-described method.

[0212] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0213] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0214] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A vector data processing method, characterized in that, Applied to a server configured with in-memory computing hardware (PIM), the method includes: The CPU in the server performs clustering processing on the input vector data, and constructs a distance lookup table based on the clustering results of the input vector data; the distance lookup table stores the pre-calculated mapping relationship between vector distances to support the calculation of the distance between the query vector and the data point; The vector processing task based on the distance lookup table is sent to the persistent PIM kernel of the memory computing hardware PIM via a message queue; the persistent PIM kernel contains multiple processing units (PUs), each of which is configured with main memory (MRAM) for caching vector data and intermediate calculation results; the vector processing task is to retrieve target vector data similar to the query vector data from the input vector data. For each of the aforementioned processing units (PUs), the query distribution data of the vector processing task is received through a hot transfer mechanism. Based on the bus ownership switching mechanism, the vector data stored in the main memory (MRAM) is used to perform vector distance calculation to obtain the distance calculation result. The distance calculation result includes the distance from the query vector data to all data points in the nearest cluster. The persistent PIM kernel sends the distance calculation results of each processing unit (PU) after sorting back to the CPU for merging processing, thereby obtaining the vector data processing result of the vector processing task.

2. The method according to claim 1, characterized in that, The process of receiving the query distribution data of the vector processing task via a hot transfer mechanism includes: The control interface link of the processing unit PU is used to construct a control path for communication between the processing unit PU and the CPU. Combining the control path and the preset data path, the system receives query and distribution data for the vector processing task; the data path is used to support the CPU to directly read and write the main memory (MRAM) via the DDR bus.

3. The method according to claim 1, characterized in that, The bus ownership switching mechanism operates in the following manner: When the processing unit (PU) is in the execution computing state, bus ownership is switched from the CPU to the processing unit (PU); the bus ownership transfer message is sent after the CPU completes the copy operation; the bus ownership includes access ownership of the main memory (MRAM); When the processing unit PU is in a state of waiting for the CPU to copy data, the ownership of the bus is switched from the processing unit PU to the CPU.

4. The method according to claim 1, characterized in that, The method further includes: During the query distribution process for the processing unit (PU), based on the selective data copying placement method, the target cluster is copied to multiple processing units (PUs) according to the usage popularity of each cluster in the clustering results; the usage popularity of the target cluster meets the preset popularity condition.

5. The method according to claim 4, characterized in that, The method further includes: By monitoring the frequency of cluster access in real time, dynamic adjustments to the data layout are triggered when changes in cluster popularity are detected. A copy of the first cluster is created on a low-load processing unit (PU), and a copy of the second cluster is deleted on a high-load processing unit (PU); the first cluster is a cluster with increasing popularity, and the second cluster is a cluster with decreasing popularity.

6. The method according to claim 4, characterized in that, The method further includes: Construct an online request distribution mapping table; the online request distribution mapping table is used to record the mapping relationship between each cluster and all copies in multiple processing units (PUs), each cluster has multiple copies, and the multiple copies are distributed in different processing units (PUs); For any cluster of replicas distributed processing units (PUs), based on the number of tasks waiting to be processed by each processing unit (PU), determine the target processing unit (PU) with the lowest current load, and send a query request for any cluster to the target processing unit (PU).

7. A vector data processing apparatus, characterized in that, An apparatus for use with a server configured with in-memory computing hardware (PIM), the apparatus comprising: The distance lookup table construction unit is configured to perform clustering processing on the input vector data via the CPU in the server, and construct a distance lookup table based on the clustering results of the input vector data; the distance lookup table stores the pre-calculated mapping relationship between vector distances to support the calculation of the distance between the query vector and the data point; A vector processing task sending unit is configured to execute a vector processing task based on the distance lookup table, which is sent via a message queue to the persistent PIM kernel of the memory computing hardware PIM. The persistent PIM kernel includes multiple processing units (PUs), each of which is configured with main memory (MRAM) for caching vector data and intermediate calculation results. The vector processing task is to retrieve target vector data similar to the query vector data from the input vector data. The distance calculation result obtaining unit is configured to execute for each of the processing units (PUs), receive query distribution data of the vector processing task through a hot transfer mechanism, perform vector distance calculation using the vector data stored in the main memory (MRAM) based on the bus ownership switching mechanism, and obtain the distance calculation result; the distance calculation result includes the distance from the query vector data to all data points in the nearest cluster; The distance calculation result feedback unit is configured to send the sorted distance calculation results of each processing unit (PU) back to the CPU for merging processing through the persistent PIM kernel to obtain the vector data processing result of the vector processing task.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the vector data processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the vector data processing method as described in any one of claims 1 to 6.

10. A computer program product, the computer program product comprising instructions, characterized in that, When the instruction is executed by the processor of the electronic device, the electronic device is able to perform the vector data processing method as described in any one of claims 1 to 6.

Images