Distributed database system based on low-difference sequence and data partitioning method

Abstract

The application belongs to the technical field of data storage and query. A distributed database system and data division method based on low-difference sequence are provided. A coordination node performs low-difference sequence-based data representative division on written data to obtain multiple data shards, and the data shards are distributed to each data node for storage and local index construction. According to a received query request, a query task is broadcast to all data nodes, local retrieval results returned by all data nodes are collected, global aggregation and sorting are performed, and the results are returned to a user or an application program. A data node receives and persistently stores data shards allocated by the coordination node, constructs a local vector index on the persistently stored data shards, performs local retrieval in response to a query task of the coordination node, and sends the retrieval results to the coordination node. The application realizes high-performance distributed database retrieval.

Description

Technical Field

[0001] This invention relates to the field of data storage and retrieval technology, specifically to a distributed database system and data partitioning method based on low-discrepancy sequences. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence, especially large-scale language models (LLMs), vectorized representation and retrieval technologies for unstructured data (such as text, images, audio, and video) have become core infrastructure. Vector databases, as systems specifically designed for storing, managing, and retrieving high-dimensional vector data, play a crucial role in areas such as similarity search, recommendation systems, and image search. Particularly in the currently popular Retrieval-Augmented Generation (RAG) application, the performance of vector databases directly determines whether large-scale language models can quickly and accurately acquire external knowledge, thereby generating high-quality, timely content. RAG systems require vector databases to achieve millisecond-level low-latency, high-throughput queries on massive (hundreds or even tens of billions) datasets, posing a significant challenge to existing technologies.

[0004] Currently, the mainstream vector database technology solutions are mainly divided into two categories: (1) Monolithic / centralized databases, such as FAISS, HNSWlib and other index libraries deployed on a single or a few high-performance servers. These systems have excellent performance when the data scale is small, but as the data volume explodes, their inherent "scale-up" mode will soon encounter performance bottlenecks; (2) Distributed databases. In order to solve the single-point performance problem, the industry has proposed a distributed solution of sharding data and storing it on multiple servers to achieve "scale-out". Common data partitioning strategies include random / hash partitioning and cluster-based partitioning.

[0005] Although existing vector database technologies have promoted the development of vector retrieval to some extent, they still have the following shortcomings when facing large-scale, high-concurrency retrieval needs, especially in scenarios with extremely high real-time requirements such as RAG:

[0006] (1) The performance and cost of centralized architecture are bottlenecks. The retrieval performance of monolithic databases is completely limited by the hardware capabilities (CPU, memory, I / O) of a single server. When the vector data scale reaches billions, the full index often cannot be fully loaded into memory. Frequent disk I / O will drastically increase query latency. Even if the performance is improved by upgrading the hardware, the cost will increase exponentially, and the performance improvement has physical limits, which cannot meet the retrieval needs of larger-scale data in the future.

[0007] (2) With the deepening of digital transformation in various industries, small and medium-sized enterprises and institutions face the need for efficient management and semantic retrieval of massive data (especially document data). However, many internal documents usually contain sensitive information and cannot rely on vector databases provided by cloud service providers. Although locally deployed distributed vector databases (such as Milvus, Qdrant, etc.) can meet the needs of large-scale retrieval, their high requirements for computing resources (such as multi-core CPU, large memory and high-speed storage) significantly increase hardware costs, making it difficult to match the financial resources of small and medium-sized enterprises and institutions. Moreover, current technology makes it difficult to build a cluster with efficient retrieval capabilities based on old office computers with various chips.

[0008] (3) Traditional distributed solutions suffer from "pseudo-parallelism" and poor performance. Random / hash partitioning leads to low query efficiency. This strategy randomly scatters data across nodes. Since the distribution of data is unrelated to the similarity of content, the relevance vector of a query request may be scattered across any node. To ensure recall, the query request must be broadcast to all data nodes. After all nodes return results, the data is aggregated and sorted. This "scatter-gather" model generates huge network overhead and communication latency when the number of nodes increases, and does not achieve true query decoupling and efficient parallelism. The clustering-based partitioning leads to representativeness bias and hotspot issues. This strategy (such as using the K-Means algorithm) divides vectors with similar content into the same partition. Although this can route queries to a few "relevant" partitions in some scenarios, it has a fundamental flaw: each partition only represents a part of the overall data, losing global representativeness. Therefore, it is impossible to obtain a globally approximate result by "querying a little bit from each partition". In addition, this method is very prone to generating "data hotspots" and "query hotspots", that is, some partitions containing popular information will be frequently accessed, causing uneven load and becoming a system bottleneck again.

[0009] (4) Existing solutions have high deployment and maintenance costs. Whether it is to scale vertically by purchasing expensive high-performance servers or to build traditional distributed clusters that require complex load balancing and query routing strategies, existing solutions have extremely high requirements for resources. This not only brings high hardware procurement costs, but also makes many enterprises that want to make use of a large number of idle ordinary computer resources in the existing network hesitate. The technical deployment threshold is too high. Summary of the Invention

[0010] To address the problems of poor data partitioning strategies, high query latency, poor scalability, and high deployment costs in the distributed architecture of existing vector databases, this invention provides a distributed database system and data partitioning method based on low-dispersion sequences. This method scientifically and effectively partitions massive high-dimensional vector data, ensuring that each data shard in the distributed storage maintains a high degree of statistical representativeness of the overall dataset. This overcomes the fundamental shortcomings in query efficiency and performance caused by traditional random partitioning or clustering. Without modifying the native code of a lightweight single-machine vector database or incurring additional hardware procurement costs, this invention utilizes existing office computers to form a heterogeneous computing cluster. Through the proposed distributed task scheduling and index sharding strategy, the retrieval load is distributed across the heterogeneous computing cluster composed of heterogeneous hardware, achieving high-performance distributed database retrieval.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a distributed database system based on low-discrepancy sequences.

[0013] A distributed database system based on low-dissimilarity sequences includes: a coordinating node and multiple data nodes communicating with the coordinating node;

[0014] The coordinating node is configured to: receive data write requests and query requests from users or applications; perform data representativeness partitioning based on low-dispersion sequences on the written data to obtain multiple data shards; distribute the data shards to various data nodes for storage and local index building; broadcast query tasks to all data nodes according to the received query requests; collect the local retrieval results returned by all data nodes; perform global aggregation and sorting; and return the results to the user or application.

[0015] Data nodes are configured to: receive and persistently store data shards allocated by the coordinating node; build local vector indexes on the persistently stored data shards; and, in response to query tasks from the coordinating node, perform local retrieval and send the retrieval results to the coordinating node.

[0016] In one implementation of the first aspect of the present invention, the coordination node includes at least: a data representativeness partitioning module; the data representativeness partitioning module includes: a vector sorting value generation submodule and a low-discrepancy sequence grouping submodule;

[0017] The vector sorting value generation submodule is configured to: map any input high-dimensional vector to a one-dimensional sortable scalar value, thereby obtaining a vector dataset sorted in ascending order of the scalar values;

[0018] The low-dissimilarity sequence grouping submodule is configured to: generate a low-dissimilarity sequence of the same length based on the total number of vectors of the written data; sort the low-dissimilarity sequence; obtain the index information of the sorted sequence; and rearrange and cut the vector dataset sorted in ascending order of scalar values ​​according to the index information to form multiple data fragments.

[0019] In one implementation of the first aspect of the present invention, the coordination node further includes: a data receiving and preprocessing module and a data distribution and database construction control module;

[0020] The data receiving and preprocessing module is configured to receive data write requests and query requests from users or applications, and perform preprocessing operations on the written data.

[0021] The data distribution and database construction control module is configured to distribute data shards to designated data nodes, and after distribution, send instructions to the corresponding data nodes so that the data nodes can persist the received data shards and build local vector indexes.

[0022] In one implementation of the first aspect of the present invention, the coordinating node further includes: a parallel retrieval control module and a result aggregation and sorting module;

[0023] The parallel retrieval control module is configured to: calculate the number of local queries that each data node needs to execute based on the received query vector and the total number of expected results K, and then broadcast the query vector and the number of local queries to all data nodes to start a parallel distributed retrieval.

[0024] The result aggregation and sorting module is configured to: collect the local search results returned by all data nodes, generate a temporary list of candidate results, sort the candidate results in the temporary list according to the similarity score or similarity distance between the candidate results and the query vector, select the top K results from the sorted list as the final results, and return them to the user or application.

[0025] In one implementation of the first aspect of the present invention, the data node includes: a data storage and indexing module and a local retrieval execution module;

[0026] The data storage and indexing module is configured to: persistently store data shards on the local disk or in memory, and build local vector indexes for the persistently stored data shards;

[0027] The local retrieval execution module is configured to: upon receiving the query vector and the number of local queries n, use the local vector index to retrieve the n vectors most similar to the query vector, along with their corresponding identification codes and similarity scores or similarity distances; package the identification codes of the n vectors and their similarity scores or similarity distances to the query vector into a local result list; and send the local result list to the coordinating node.

[0028] In one possible implementation of the first aspect of the present invention, a decentralized architecture is adopted, wherein the coordinating node is one of a plurality of data nodes, and any data node can be transformed into a coordinating node and perform the functions of the coordinating node.

[0029] In one possible implementation of the first aspect of the present invention, the coordinating node is an office computer of various types, and a heterogeneous computing cluster with multiple coordinating nodes is formed by the various types of office computers.

[0030] In one possible implementation of the first aspect of the present invention, a coordination node cluster is constructed, which contains multiple coordination nodes. Each coordination node is connected to the other through a consensus protocol to achieve master-slave switching of the multiple coordination nodes.

[0031] Secondly, the present invention provides a data partitioning method based on low-discrepancy sequences.

[0032] A data partitioning method based on low-dispersion sequences includes the following steps:

[0033] Obtain the original dataset containing N high-dimensional vectors;

[0034] Generate a one-dimensional sorting value for each high-dimensional vector in the original dataset D, and sort the original dataset D in ascending order according to the one-dimensional sorting values ​​to obtain dataset D'.

[0035] Based on the total number of vectors N in the original dataset D, a one-dimensional low-dissimilarity sequence X of length N is generated. The one-dimensional low-dissimilarity sequence X is sorted, and the index of the original position of each element before sorting is recorded. Finally, a sorted index R of length N is obtained.

[0036] Based on the generated sorting index R, the sorted dataset D' is rearranged to obtain a new dataset D'';

[0037] When the number of data nodes is K, the size of each data partition is set to M=N / K, and the dataset D'' is sequentially divided into K data partitions of size M.

[0038] The generated K data shards are sent to the K corresponding data nodes or to a cluster consisting of multiple independent vector database instances that only provide basic retrieval services.

[0039] In one implementation of the second aspect of the present invention, generating a one-dimensional sorting value for each high-dimensional vector in the original dataset D includes:

[0040] One-dimensional ranking values ​​can be obtained by using principal component analysis; or by using a manifold learning algorithm; or by using an autoencoder; or by using feature weighting; or by directly selecting the value of the dimension that best represents the data distribution characteristics as the one-dimensional ranking value.

[0041] The Hua Luogeng-Wang Yuan method based on irrational numbers can be used to generate low-dissimilarity sequences; alternatively, Halton sequences can be used as low-dissimilarity sequences; alternatively, Sobol sequences can be used as low-dissimilarity sequences; alternatively, Faure sequences can be used as low-dissimilarity sequences.

[0042] Thirdly, the present invention provides a distributed parallel retrieval method.

[0043] A distributed parallel retrieval method, utilizing the distributed database system based on low-discrepancy sequences as described in the first aspect of this invention, includes the following processes:

[0044] The coordinating node receives query requests from users or applications, the query requests including query vectors and the total number of results expected to be returned;

[0045] The coordinating node determines the number K of data nodes participating in this query, calculates the number n of local search results that each data node needs to return, and broadcasts the query vector Q and the calculated number of local search results n to all data nodes.

[0046] After receiving the query vector and the number of local queries n, the data node uses the local vector index to retrieve the query vector that is most similar to the query vector. The system takes n vectors and their corresponding identification codes and similarity scores or similarity distances, packages the identification codes of the n vectors and their similarity scores or similarity distances with the query vector into a local result list, and sends the local result list to the coordinating node.

[0047] The coordinating node reorders all candidate vectors based on the received list of all local results, and selects the top K vectors as the final search results.

[0048] In one implementation of the third aspect of the present invention, an approximate nearest neighbor algorithm is used to retrieve the vector most similar to the query vector. For a vector, the approximate nearest neighbor algorithm can be: a quantization-based method; or a tree-based method; or a graph-based method; or a hash-based method.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] This invention provides an economical and feasible large-scale vector database deployment scheme, which improves the system's resource utilization and reduces deployment costs. By scientifically splitting massive amounts of data into multiple small-scale, highly representative shards, the entire system can run on a cluster of general-purpose computers with ordinary performance, small hard drives, and low memory. This effectively avoids dependence on expensive high-performance servers and significantly reduces hardware procurement and system maintenance costs. The database architecture of this invention can make full use of a large number of existing devices with idle computing capabilities in the enterprise network, achieving effective resource utilization and removing cost barriers for the popularization and application of large-scale vector retrieval technology.

[0051] This invention provides an economical and feasible large-scale vector database deployment scheme. Through innovative architecture design, it can make full use of the office computers of small and medium-sized enterprises and institutions to form a heterogeneous computing cluster, bridging the gap between resource efficiency and scalability. Without modifying the native code of the lightweight single-machine vector database or increasing the cost of purchasing new hardware, it can utilize existing office computers (including various domestically produced computers based on domestically developed chips) to form a heterogeneous computing cluster. Through innovative distributed task scheduling and index sharding strategies, the retrieval load is distributed to the office computer cluster composed of heterogeneous hardware, realizing high-performance distributed database retrieval.

[0052] This invention uses a low-dissimilarity sequence (LDS) based on number theory for data sharding, ensuring that each data shard is a high-quality "microcosm" of the overall data, thus achieving effective decomposition and parallelization of computational tasks. Based on data sharding, a global query task that could not be executed on a single ordinary device due to excessive computational and memory requirements can be successfully decomposed into multiple independent sub-query tasks that are executed in parallel on all shards. This database architecture fundamentally solves the single-point computation bottleneck problem of centralized systems.

[0053] Although the vector database of this invention runs on low-configuration hardware, since each data shard is highly representative of the whole, only a small number of the most relevant results need to be retrieved from each shard for aggregation. This yields Top-K results that are highly consistent with the results of global retrieval on a high-performance server. High-quality retrieval results are guaranteed under low-cost deployment, ensuring that the economy of the solution is not at the expense of retrieval quality. It can meet the stringent requirements for result accuracy in application scenarios such as RAG.

[0054] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0055] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0056] Figure 1 A schematic diagram of the architecture of a distributed database system based on low-discrepancy sequences provided as an exemplary embodiment of the present invention;

[0057] Figure 2 A schematic diagram of the functional modules of a distributed database system based on low-discrepancy sequences provided as an exemplary embodiment of the present invention;

[0058] Figure 3 A flowchart illustrating a data partitioning method based on low-discrepancy sequences, provided as an exemplary embodiment of the present invention;

[0059] Figure 4 Provided as an exemplary embodiment of the present invention Figure 3 A detailed flowchart of step S2 is shown below;

[0060] Figure 5 Provided as an exemplary embodiment of the present invention Figure 3 A detailed flowchart of step S4 is provided.

[0061] Figure 6 This is a flowchart illustrating a distributed parallel retrieval method provided as an exemplary embodiment of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0064] As described in the background section, existing technologies suffer from fundamental contradictions in achieving efficient, low-cost, and scalable retrieval of large-scale vector data. Therefore, this implementation proposes a novel method for vector data partitioning and organization, along with a distributed retrieval system based on this method, aiming to overcome the limitations of traditional solutions in hardware deployment and cost-effectiveness. Specifically, this invention scientifically and effectively partitions a vector database that is too large to run on a single ordinary server into multiple data shards, ensuring that each shard is a statistically highly representative microcosm of the overall dataset. This approach enables the entire system to be deployed on a distributed cluster composed of low-cost, low-configuration general-purpose hardware, thereby fundamentally solving the problems of high hardware costs and high deployment barriers in existing technologies while maintaining high-quality retrieval performance, and fully utilizing existing idle computing resources in the network.

[0065] At the system level, this invention provides a distributed database system based on low-discrepancy sequences. The distributed vector database consists of one or more coordinating nodes and multiple data nodes, which are interconnected through a network. The coordinating nodes are responsible for managing metadata, performing data partitioning tasks, and coordinating the retrieval process; the data nodes are responsible for storing data fragments and performing local retrieval.

[0066] At the methodological level, this invention includes two core processes: data representativeness partitioning and database construction process and distributed parallel retrieval process.

[0067] The data representativeness partitioning and database construction process of this invention abandons the traditional random partitioning or content similarity-based clustering strategies, and creatively adopts a low-discrepancy sequence based on number theory and statistics as the grouping criterion. Through this process, massive vector datasets are divided into multiple data fragments, ensuring that each fragment is a highly representative sample of the overall dataset in a statistical sense, thus achieving the effect of "minimizing inter-group differences and maximizing intra-group differences".

[0068] Based on the unique partitioning results described above, the distributed parallel retrieval process of this invention can efficiently decompose the global retrieval task into parallel execution on all data nodes. Since each data shard is approximately a "microcosm" of the overall data, it is only necessary to retrieve a small number of the most relevant results from each node for aggregation to approximate the final result of the global retrieval with high quality.

[0069] In summary, this invention constructs a distributed vector database system that truly supports efficient parallel retrieval in its architecture through a novel data organization method. This enables large-scale, high-precision vector retrieval on low-cost, general-purpose hardware, and also provides superior scalability.

[0070] like Figure 1The diagram shown is a schematic of a distributed database system based on low-discrepancy sequences provided by the present invention, which includes three core components: a coordinating node, data nodes, and a network connecting them.

[0071] The coordinating node is the "brain" and central entry point of the entire distributed system. It can be a standalone server or a highly available cluster of multiple servers. Its main responsibilities include:

[0072] Request processing: As the unified interface of the system, it receives data write requests and query requests from users or applications;

[0073] Data partitioning: The core technology of this invention, namely "data representative partitioning based on low dissimilarity sequences", divides the received massive vector data into multiple data fragments;

[0074] Metadata management: Maintain the metadata information of the entire cluster, including the address and status of data nodes, the data sharding information stored on each node, and the summary information of the global index, etc.

[0075] Task distribution and coordination: Distribute the divided data shards to each data node for storage and local index building. When processing query requests, broadcast the query task to all (or specified) data nodes and coordinate the parallel retrieval process.

[0076] Result aggregation: Collect the local search results returned by all data nodes, perform global aggregation and sorting, and return the final results to the user.

[0077] Data nodes are the units responsible for data storage and computation execution in a distributed system. A system typically consists of a cluster of multiple data nodes, and the cluster size can be dynamically scaled according to data volume and query load. Their main responsibilities include:

[0078] Data storage: Receives and persistently stores data shards allocated by the coordinating node. Each data node typically manages one or more shards.

[0079] Local index building: The stored data is sharded, and efficient approximate nearest neighbor (ANN) algorithms such as HNSW and IVF are used to build local vector indexes to accelerate local queries;

[0080] Local retrieval: In response to a query command from the coordinating node, perform a vector similarity search on the local index and return a specified number of local results that are most similar to the query vector;

[0081] Node Scalability: The number of data nodes can be horizontally expanded as needed. Adding new nodes can share the storage and query pressure, thereby linearly improving the capacity and performance of the entire system. As a key advantage of this invention, the solution has low hardware configuration requirements for data nodes. These data nodes can be dedicated servers or ordinary computers with idle computing resources in an enterprise's internal network, thus effectively utilizing existing hardware resources and achieving low-cost deployment.

[0082] A network is a channel connecting the coordinating node with all data nodes, as well as facilitating communication between nodes. It is typically a standard TCP / IP Ethernet network. Network bandwidth and latency are key factors affecting the overall performance of the system, especially during the data distribution and query result aggregation stages.

[0083] To achieve the technical solution of this invention, the vector database system of this invention is logically divided into a series of functional modules. These modules are distributed on coordination nodes and data nodes, working together to complete data partitioning, database construction, and retrieval, such as... Figure 2 As shown.

[0084] The functional modules mainly include: a data receiving and preprocessing module, a data representativeness partitioning module, a data distribution and database construction control module, a parallel retrieval control module, a result aggregation and sorting module on the coordination node, and a data storage and indexing module and a local retrieval execution module on the data node.

[0085] The data receiving and preprocessing module serves as the entry point for data inflow, responsible for receiving batches of high-dimensional vector datasets from external users or systems. This module performs preliminary preprocessing operations such as format validation, cleaning, and normalization on the data, preparing standardized data for subsequent partitioning steps.

[0086] The data representativeness partitioning module is responsible for executing innovative data partitioning algorithms to divide the overall dataset into multiple data pieces with high statistical representativeness. It consists of a vector sorting value generation submodule and a low dissimilarity sequence (BDS) grouping submodule.

[0087] The vector sorting value generation submodule is responsible for mapping the input high-dimensional vector into a one-dimensional, sortable scalar value that reflects the distribution characteristics of the vector along a certain main dimension.

[0088] In a preferred embodiment of the present invention, the module employs the Principal Component Analysis (PCA) method to calculate the first principal component of the data and uses the projection value of each vector onto the principal component as its one-dimensional ranking value.

[0089] Optionally, in other implementations, any method that can generate stable and discriminative one-dimensional scalar values ​​for high-dimensional vectors falls within the scope of this invention, such as using manifold learning algorithms like UMAP and t-SNE to reduce the output to one dimension, or other feature selection and projection functions.

[0090] The Low Dissimilarity Sequence (BDS) grouping submodule is responsible for generating the results of the submodule based on the vector sorting values. Combining the low dissimilarity sequence theory, it groups the entire data. Specifically, this includes: first, generating a low dissimilarity sequence of appropriate length based on the total number of vectors N in the dataset; then, obtaining the index information of the sorted sequence; and finally, rearranging and cutting the vector dataset that has been sorted by sorting value according to this index information, ultimately forming K data partitions. This process ensures that each partition is a uniform and unbiased sample of the overall data distribution.

[0091] The data distribution and database control module is responsible for distributing the data fragments generated by the "data representativeness partitioning module" to the designated data nodes via the network. After the distribution is completed, the module will send instructions to the corresponding data nodes, requiring them to persist the received data fragments and build local vector indexes.

[0092] The parallel retrieval control module is responsible for processing user query requests. When it receives a query vector Q and the expected total number of results K, the module calculates the number of results n that each data node needs to return from its local query (for example, n is equal to the ratio of K to the number of nodes). Then, it broadcasts the query vector Q and the parameter n to all data nodes in the cluster, thereby initiating a parallel distributed retrieval.

[0093] The results aggregation and sorting module is responsible for collecting the local search results returned by all data nodes. Since each node returns n most similar results, this module will obtain a result containing "n The system first generates a temporary list of candidate results with a "number of nodes". Then, the result aggregation and sorting module performs a global and precise sorting based on the original similarity score (or distance) between these candidate results and the query vector, and selects the top K results from the sorted list as the final results, which are then returned to the user.

[0094] The data storage and indexing module is responsible for receiving, managing, and indexing data shards allocated by the coordinating node. This module persistently stores the data shards on the local disk or in memory and uses efficient Approximate Nearest Neighbor (ANN) search algorithms (such as HNSW, IVF-PQ, LSH, etc.) to build local index files for these data. This index is only for local data shards, is small in size, and has a fast build and load speed.

[0095] The local retrieval execution module is responsible for executing specific retrieval tasks from the coordinating node. After receiving the query vector Q and the number of local queries n, this module will use the local index built by the "data storage and indexing module" to quickly retrieve the n vectors most similar to Q, along with their corresponding IDs (i.e., identification codes) and similarity scores, and return this list of results to the coordinating node via the network.

[0096] It should be noted that the above-described scheme of the present invention describes a centralized coordination architecture consisting of coordination nodes and data nodes. However, the core partitioning and retrieval ideas of the present invention are also applicable to other distributed architectures.

[0097] For example, an optional high-availability coordinator node architecture can be adopted. To avoid single points of failure of the coordinator node, a master-slave mode can be used, or a coordinator node cluster can be built based on the Raft / Paxos consensus protocol by introducing components such as ZooKeeper and etcd to achieve high availability.

[0098] For example, alternatively, a decentralized / peer-to-peer (Peer-to-Peer) architecture can be adopted. In certain application scenarios, the centralized role of the coordinating node can be weakened. Each node acts as a data node, storing data shards, and also undertakes some metadata synchronization and query routing coordination functions through peer-to-peer network protocols such as Gossip. Query requests can be sent to any node in the network, which can then initiate broadcasting and result aggregation.

[0099] Alternatively, a separate architecture for data partitioning and retrieval services can be adopted. The data partitioning function of this invention can be implemented as an independent, offline "data preprocessing service". This service is responsible for periodically receiving new data, performing representative partitioning, and pushing the generated data shards to a cluster consisting of multiple independent vector database instances that only provide basic retrieval services.

[0100] The database system of this invention can be deployed not only on server clusters, but also flexibly deployed in various modern computing environments, such as:

[0101] It can be deployed on public or private clouds using virtual machines or containers, and managed, scaled and recovered automatically through container orchestration tools; alternatively, the coordination node (or high availability cluster) can be deployed on a high-performance central server, while data nodes are widely deployed on a large number of ordinary-performance computers, workstations or even edge computing devices in the enterprise network to form a heterogeneous hybrid computing cluster.

[0102] Based on the aforementioned distributed vector database system based on low-dispersion sequences, this implementation proposes a data partitioning method based on low-dispersion sequences. This method describes how the invention processes a large-scale original vector dataset into multiple highly statistically representative data partitions using an innovative partitioning approach, and ultimately completes index construction on various data nodes of the distributed system. This process is mainly scheduled and executed by a coordinating node. Specifically, as follows... Figure 3 As shown, the process includes the following:

[0103] Step S1: Receive the raw dataset.

[0104] The process begins with the data receiving and preprocessing module of the coordination node receiving a raw dataset containing N high-dimensional vectors. In this step, vectors can be preprocessed, such as by normalization, to ensure data consistency.

[0105] Step S2: Generate one-dimensional sort values ​​and perform preliminary sorting of the data.

[0106] like Figure 4 As shown, in order to subsequently group based on a one-dimensional sequence, the vector sorting value generation submodule in the data representativeness partitioning module of the coordination nodes processes each high-dimensional vector in the dataset D. Process it to generate a corresponding one-dimensional scalar sort value. .

[0107] One feasible approach is to perform this through Principal Component Analysis (PCA). First, the first principal component axis of the entire dataset D is calculated, and then each vector... Projected onto this axis, its projection value is the one-dimensional sort value of the vector. After processing, a one-dimensional sorted value sequence is obtained that corresponds one-to-one with the vectors in D. Subsequently, the original dataset D is sorted in ascending (or descending) order according to the values ​​of sequence E, resulting in a pre-sorted dataset. ,in It is based on its sort value The vector after arrangement.

[0108] Alternatively, other dimensionality reduction methods can be used in other implementations. For example, algorithms such as UMAP (Uniform Manifold Approximation and Projection) and t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used to reduce the dimensionality to one dimension as the ranking value. Alternatively, an autoencoder model can be trained, and the one-dimensional output of its bottleneck layer can be used as the ranking value for each vector. Or, for vectors in certain domains, a weighted sum of specific dimensions in the vector can be performed based on prior knowledge, or the dimension that best represents the data distribution characteristics can be directly selected as the ranking value.

[0109] Step S3: Generate low-dissimilarity sequences (BDS) and their sorting index.

[0110] The Low Dissimilarity Sequence (BDS) grouping submodule in the data representativeness partitioning module of the coordinating node generates a one-dimensional low dissimilarity sequence of length N based on the total number of vectors N in the dataset. .

[0111] A low-dispersion sequence is a deterministic sequence that is more uniformly distributed in space than a pseudo-random sequence. In one embodiment of the present invention, this sequence can be generated using the Hua Luogeng-Wang Yuan method. For example, taking an irrational number α (such as π), the i-th element of the sequence... can be The calculation yielded the result.

[0112] It should be noted that other deterministic or quasi-random sequence generation algorithms with good uniform distribution properties are also applicable, such as:

[0113] Halton sequence: A sequence generated using coprime integers as the base;

[0114] Sobol sequence: Another widely used low-dissimilarity sequence;

[0115] Faure sequence: Constructed based on the van der Koppit sequence at a specific cardinality.

[0116] After generating sequence X, sort it and record the index of each element's original position before sorting. This results in a permutation index of length N. ,in This indicates that the element at position i after sorting comes from the element at position i in the original sequence X. The index R is key to achieving representative grouping.

[0117] Step S4: Perform representative grouping based on the sort index.

[0118] like Figure 5 As shown, the coordinating node uses the sorting index R generated in step S3 to rearrange and group the dataset D' obtained in the preliminary sorting in step S2. Specifically, a new empty dataset D'' is created, and vectors are taken from D' and placed into D'' according to the order of index R. That is, the i-th vector in D'' is the i-th vector in D''. This rearrangement operation breaks the local clustering that may occur in step S2 due to content similarity (similar projection values ​​on principal components). Through the uniform distribution characteristics of BDS, vectors with different features are evenly distributed throughout the D'' sequence.

[0119] If the number of data nodes in the system is set to K, then the size of each data partition is M = N / K. The rearranged dataset D'' is then sequentially divided into K data partitions of size M. , , ..., .Right now , ,..., .

[0120] After this step, each data fragment is obtained The internal vectors have significant differences (from) Different regions), and different partitions and The overall data distributions are highly similar across the datasets, and each slice constitutes a high-quality, unbiased representative sample of the original dataset D.

[0121] Step S5: Distribute data fragments.

[0122] The data distribution and database creation control module of the coordination node initiates the data distribution task, dividing the K data fragments generated in step S4 into... , ,..., It is then sent to K corresponding data nodes via the network.

[0123] Step S6: Build local indexes in parallel.

[0124] After receiving the data shard assigned to it, each data node's data storage and indexing module begins operation. This module persistently stores the data shards and independently and in parallel uses ANN algorithms such as HNSW to build vector indexes for its local data. This step has dual advantages: fast construction speed, as each node processes only 1 / K of the total data, making index building very rapid; and low memory consumption, meaning the peak memory consumption required for index building is significantly reduced. This solves the memory overflow problem caused by building indexes for massive amounts of data at once in a monolithic architecture, allowing the entire database construction process to be successfully executed on low-configuration devices with limited memory resources. After all nodes have completed index building, the entire distributed vector database construction process ends, and the system enters a queryable state.

[0125] Based on the aforementioned distributed vector database system based on low-dissimilarity sequences, this implementation proposes a distributed parallel retrieval method. It describes how, after receiving a user's query request, the nodes in the distributed vector database system collaborate to efficiently complete the query task on a distributed architecture and accurately return the most similar vector results. The core of this process lies in leveraging the high representativeness of each data shard to the overall data, successfully decomposing a large-scale global search task that would otherwise be impossible to execute on a single low-configuration device into multiple small-scale local search tasks that can be processed in parallel on each node. Specifically, for example... Figure 6 As shown, it includes the following steps:

[0126] Step T1: Receive query request.

[0127] The process begins with the parallel retrieval control module of the coordinating node receiving a query request from the user or application. This request typically contains two main parameters:

[0128] Query vector Q: A high-dimensional vector with the same dimensions as the vectors stored in the database;

[0129] The expected total number of results, K: the number of most similar vectors the user hopes to obtain.

[0130] Step T2: Calculate the number of local searches and broadcast the query.

[0131] The parallel retrieval control module of the coordinating node performs the following operations based on the query request and the current system status:

[0132] Determine the number of active data nodes K_active participating in this query. Under normal circumstances, K_active is equal to the total number of data nodes K in the system, in order to maximize the utilization of parallel capabilities.

[0133] Calculate the number of local search results, n, that each data node needs to return. A preferred calculation is n = ceil(K / K_active), where ceil is the floor function. This ensures that even in the worst case, the total number of candidate results collected is at least K. In practice, to improve the accuracy of the final results, n can also be set to a value slightly larger than K / K_active, for example... , where α is a redundancy factor greater than 1 (such as 1.1 or 1.2).

[0134] The query vector Q and the calculated number of local searches n are broadcast over the network to all K_active active data nodes.

[0135] Step T3: Each data node performs local retrieval in parallel.

[0136] Upon receiving a query instruction from the coordinating node, each data node in the cluster immediately, independently, and in parallel performs the following operations on its local retrieval execution module:

[0137] Perform an approximate nearest neighbor (ANN) search on the vector index corresponding to the data shards stored locally.

[0138] Find the n vectors that have the highest similarity (or the closest distance) to the query vector Q.

[0139] Pack the IDs of these n vectors, along with their similarity scores (or distances) to the query vector Q, into a local results list.

[0140] It should be noted that this invention does not impose specific restrictions on the Approximate Nearest Neighbor (ANN) search algorithm used locally by data nodes; any efficient ANN indexing algorithm can be used as an alternative, including but not limited to:

[0141] Quantification-based methods: such as IVF-PQ (Inverted File with Product Quantization);

[0142] Tree-based methods: such as Annoy (Approximate Nearest Neighbors Oh Yeah);

[0143] Graph-based methods (preferred): such as HNSW (Hierarchical Navigable Small World), are employed in preferred embodiments;

[0144] Hash-based methods, such as LSH (Locality-Sensitive Hashing).

[0145] Step T4: Collect all local results.

[0146] After each data node completes its local retrieval, it sends its generated local result list back to the coordinating node via the network. The coordinating node's result aggregation and sorting module is responsible for listening to and receiving the results returned by all data nodes.

[0147] Step T5: Global aggregation and reordering.

[0148] After receiving the results from all data nodes (or within the set timeout period), the result aggregation and sorting module of the coordination node performs the following operations:

[0149] List all local results (total) (The individual candidate results) are merged into a single global candidate result list.

[0150] For all vectors in this global candidate list, a global and precise re-ranking is performed based on their similarity score (or distance) to the query vector Q. This step eliminates the ranking bias that may be caused by differences in the local data distribution of each node, ensuring the accuracy of the final result.

[0151] Step T6: Select the global Top-K and return.

[0152] From the sorted global candidate result list in step T5, the top K vectors are selected as the final search results, and the coordinating node returns this final result list to the requesting user or application. This completes one distributed parallel search process.

[0153] The distributed vector database system, data partitioning method, and retrieval method based on low-dispersion sequences provided by this invention have the following significant and fundamental advantages in solving the practical deployment challenges of large-scale vector retrieval:

[0154] (1) It greatly reduces the hardware threshold and deployment cost, and realizes the reuse of existing resources.

[0155] The core advantage of this invention lies in its ability to enable the deployment and operation of large-scale vector databases on clusters composed of low-cost, low-configuration general-purpose hardware. Specifically, this includes:

[0156] Solving the single-point hardware bottleneck: In the background technology, monolithic databases have extremely high requirements for the memory, CPU and storage performance of a single server when facing massive amounts of data, and the hardware cost can easily reach hundreds of thousands or even millions of yuan; This invention, through scientific data partitioning, decomposes a large-scale database, for example, that requires hundreds of GB of memory to load indexes, into multiple small database shards that only require a few GB of memory. This fundamentally solves the problem of deployment due to hardware limitations.

[0157] This invention enables the effective utilization of idle resources: It has extremely low hardware configuration requirements for data nodes, allowing them to be flexibly deployed on a large number of existing ordinary servers or personal computers with idle computing and storage capabilities in enterprise networks. This revitalizes dormant IT assets, achieves effective resource utilization, avoids large-scale hardware procurement for a single application scenario, and is extremely cost-effective.

[0158] (2) Under low-cost deployment, the retrieval accuracy is highly consistent with that of centralized solutions.

[0159] This invention achieves low-cost deployment without sacrificing search quality. Its unique representativeness classification method ensures high accuracy of search results, specifically including:

[0160] High-quality global result approximation: The partitioning based on low-dispersion sequences ensures that each data slice is a high-quality "microcosm" of the overall dataset in a statistical sense. Therefore, the locally optimal results retrieved from each slice can form a high-quality candidate set covering the global data distribution after aggregation, thus ensuring the accuracy of the final result.

[0161] Experimental data support: In the comparative experiment conducted on a certain test dataset, the following metrics were used to measure the quality of the search results:

[0162] Match Rate: This refers to the proportion of K results returned whose category matches the original category of the query text. This metric measures the overall relevance of the search results, and its calculation method is as follows:

[0163] Match rate = (Number of category matches in the returned results / Total number of returned results K) × 100%;

[0164] First Result Distance: This refers to the distance value among the K returned results that is most similar to the query vector (i.e., the smallest distance). This metric measures the system's ability to find the best match. The smaller the distance value, the higher the matching degree.

[0165] Average Distance: This refers to the arithmetic mean of the distances between the K returned results and the query vector. This metric reflects the overall similarity level of the returned result set.

[0166] Under the condition of retrieving the top-10 results, the experimental results show that:

[0167] The distributed retrieval scheme of this invention achieves an average matching rate of 50.50%, while the traditional centralized retrieval scheme running on a single high-performance server achieves an average matching rate of 51.21%, with a difference of less than 1 percentage point.

[0168] The two are completely consistent in terms of the mean distance of the first result. In terms of the mean distance of all results, the mean distance using the present invention is 0.3025, and the mean distance of the centralized query is 0.2876, which is also very small.

[0169] Experimental data strongly demonstrates that while reducing hardware costs by several orders of magnitude, the accuracy of its retrieval results is almost indistinguishable from that of expensive centralized solutions, successfully achieving the goal of "low cost and high quality".

[0170] (3) The present invention realizes the effective decomposition of computational tasks, providing a solid foundation for system scalability.

[0171] This invention provides a perfect architectural foundation for the horizontal scaling of the system by decomposing a massive computational task into multiple parallel subtasks, specifically including:

[0172] Decomposition of computational and storage pressure: A global retrieval or index building task that was originally impossible to execute on a single ordinary device due to excessive computational and memory requirements was successfully decomposed into multiple independent subtasks that were executed in parallel on all shards. This not only solved the "can it run?" problem, but also means that when the amount of data or concurrent requests increases, the system's capacity can be linearly expanded by simply adding ordinary hardware nodes;

[0173] Reduced single-point performance bottleneck: Although the total end-to-end time of a distributed solution may be higher than that of an ideal single-machine memory computing due to network communication overhead, this invention eliminates the dependence on single-point high-performance computing; in many business scenarios, "being able to complete it in 1 minute with a low-cost solution" is far more practical and feasible than "having to spend a lot of money to complete it in 10 seconds".

[0174] (4) This invention promotes the popularization of large-scale, high-precision AI applications.

[0175] The low cost and high quality of this invention significantly lower the deployment threshold for advanced AI applications such as Retrieval Augmentation (RAG), making large-scale RAG applications possible. In existing solutions, building a vector knowledge base for a RAG system often requires terabytes of memory and high-end CPU clusters, which is prohibitively expensive and deters many enterprises. This invention provides an economical and feasible alternative, enabling enterprises to deploy large-scale knowledge bases that were previously unaffordable using existing IT resources. This represents a qualitative leap from "too costly to do" to "now economically feasible," greatly promoting the popularization and application of technologies such as RAG.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed database system based on low-dispersion sequences, Its features are, include: The coordinating node and multiple data nodes that communicate with the coordinating node; The coordinating node is configured to: receive data write requests and query requests from users or applications; perform data representativeness partitioning based on low-dispersion sequences on the written data to obtain multiple data shards; distribute the data shards to various data nodes for storage and local index building; broadcast query tasks to all data nodes according to the received query requests; collect the local retrieval results returned by all data nodes; perform global aggregation and sorting; and return the results to the user or application. Data nodes are configured to: receive and persistently store data shards allocated by the coordinating node; build local vector indexes on the persistently stored data shards; and, in response to query tasks from the coordinating node, perform local retrieval and send the retrieval results to the coordinating node. The coordination node includes at least: a data representativeness partitioning module; the data representativeness partitioning module includes: a vector sorting value generation submodule and a low-discrepancy sequence grouping submodule; The vector sorting value generation submodule is configured to: map any input high-dimensional vector to a one-dimensional sortable scalar value, thereby obtaining a vector dataset sorted in ascending order of the scalar values; The low-dissimilarity sequence grouping submodule is configured to: generate a low-dissimilarity sequence of the same length based on the total number of vectors of the written data; sort the low-dissimilarity sequence; obtain the index information of the sorted sequence; and rearrange and cut the vector dataset sorted in ascending order of scalar values ​​according to the index information to form multiple data fragments. The coordination node further includes: a parallel retrieval control module; The parallel retrieval control module is configured to: calculate the number of local queries that each data node needs to execute based on the received query vector and the total number of expected results K, and then broadcast the query vector and the number of local queries to all data nodes to start a parallel distributed retrieval.

2. The distributed database system based on low-discrepancy sequences as described in claim 1, characterized in that, The coordination node also includes: a data receiving and preprocessing module and a data distribution and database construction control module; The data receiving and preprocessing module is configured to receive data write requests and query requests from users or applications, and perform preprocessing operations on the written data. The data distribution and database construction control module is configured to distribute data shards to designated data nodes, and after distribution, send instructions to the corresponding data nodes so that the data nodes can persist the received data shards and build local vector indexes.

3. The distributed database system based on low-discrepancy sequences as described in claim 1, characterized in that, The coordination node also includes: a result aggregation and sorting module; The result aggregation and sorting module is configured to: collect the local search results returned by all data nodes, generate a temporary list of candidate results, sort the candidate results in the temporary list according to the similarity score or similarity distance between the candidate results and the query vector, select the top K results from the sorted list as the final results, and return them to the user or application.

4. The distributed database system based on low-dispersion sequences as described in claim 1, characterized in that, The data node includes: a data storage and indexing module and a local retrieval execution module; The data storage and indexing module is configured to: persistently store data shards on the local disk or in memory, and build local vector indexes for the persistently stored data shards; The local retrieval execution module is configured to: upon receiving the query vector and the number of local queries n, use the local vector index to retrieve the n vectors most similar to the query vector, along with their corresponding identification codes and similarity scores or similarity distances; package the identification codes of the n vectors and their similarity scores or similarity distances to the query vector into a local result list; and send the local result list to the coordinating node.

5. The distributed database system based on low-discrepancy sequences as described in claim 1, characterized in that, The architecture is decentralized, and the coordinating node is one of multiple data nodes. Any data node can be transformed into a coordinating node and perform the functions of the coordinating node. Alternatively, a coordination node cluster can be constructed, which contains multiple coordination nodes. These coordination nodes communicate and connect through a consensus protocol to achieve master-slave failover among the multiple coordination nodes. Alternatively, a coordination node cluster can be constructed, with various types of office computers forming the coordination node cluster.

6. A data partitioning method based on low-dissimilarity sequences, characterized in that, Includes the following processes: Obtain the original dataset containing N high-dimensional vectors; Generate a one-dimensional sorting value for each high-dimensional vector in the original dataset D, and sort the original dataset D in ascending order according to the one-dimensional sorting values ​​to obtain dataset D'. Based on the total number of vectors N in the original dataset D, a one-dimensional low-dissimilarity sequence X of length N is generated. The one-dimensional low-dissimilarity sequence X is sorted, and the index of the original position of each element before sorting is recorded. Finally, a sorted index R of length N is obtained. Based on the generated sorting index R, the sorted dataset D' is rearranged to obtain a new dataset D''; When the number of data nodes is K, the size of each data partition is set to M=N / K, and the dataset D'' is sequentially divided into K data partitions of size M. The generated K data shards are sent to the K corresponding data nodes or to a cluster consisting of multiple independent vector database instances that only provide basic retrieval services.

7. The data partitioning method based on low-dissimilarity sequences as described in claim 6, characterized in that, Generate a one-dimensional sort value for each high-dimensional vector in the original dataset D, including: One-dimensional ranking values ​​can be obtained by using principal component analysis; or by using a manifold learning algorithm; or by using an autoencoder; or by using feature weighting; or by directly selecting the value of the dimension that best represents the data distribution characteristics as the one-dimensional ranking value. The Hua Luogeng-Wang Yuan method based on irrational numbers can be used to generate low-dissimilarity sequences; alternatively, Halton sequences can be used as low-dissimilarity sequences; alternatively, Sobol sequences can be used as low-dissimilarity sequences; alternatively, Faure sequences can be used as low-dissimilarity sequences.

8. A distributed parallel retrieval method, characterized in that, The distributed database system based on low-dispersion sequences according to any one of claims 1-5 includes the following process: The coordinating node receives query requests from users or applications, the query requests including query vectors and the total number of results expected to be returned; The coordinating node determines the number K of data nodes participating in this query, calculates the number n of local search results that each data node needs to return, and broadcasts the query vector Q and the calculated number of local search results n to all data nodes. After receiving the query vector and the number of local queries n, the data node uses the local vector index to retrieve the query vector that is most similar to the query vector. The system takes n vectors and their corresponding identification codes and similarity scores or similarity distances, packages the identification codes of the n vectors and their similarity scores or similarity distances with the query vector into a local result list, and sends the local result list to the coordinating node. The coordinating node reorders all candidate vectors based on the received list of all local results, and selects the top K vectors as the final search results.

9. The distributed parallel retrieval method as described in claim 8, characterized in that, The approximate nearest neighbor algorithm is used to retrieve the vector most similar to the query vector. For a vector, the approximate nearest neighbor algorithm is: a quantization-based method; Alternatively, a tree-based method; or a graph-based method; or a hash-based method.

Images