Distributed database system based on low-difference-degree sequence and data partitioning method

Abstract

The invention belongs to the technical field of data storage and query. According to the distributed database system based on the low-difference-degree sequence and the data partitioning method, a coordination node executes data representative partitioning based on the low-difference-degree sequence on written data to obtain a plurality of data fragments, and the data fragments are distributed to all data nodes for storage and construction of a local index; broadcasting a query task to all the data nodes according to a received query request, collecting local retrieval results returned by all the data nodes, performing global aggregation and sorting, and returning the results to a user or an application program; and the data node receives and persistently stores the data fragments distributed by the coordination node, constructs a local vector index for the persistently stored data fragments, performs local retrieval in response to a query task of the coordination node, and sends a retrieval result to the coordination node. According to the method, high-performance distributed database retrieval is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage and query, and particularly relates to a distributed database system based on low-difference sequences and a data division method. BACKGROUND

[0002] The statements in this section merely provide background technology related to the present application and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence, especially large language models (LLM), the vectorization representation and retrieval technology of unstructured data (such as text, pictures, audio and video) has become a core infrastructure. As a system specially designed to store, manage and retrieve high-dimensional vector data, the vector database plays a key role in similarity search, recommendation system, image search, etc. Especially in the current popular retrieval-augmented generation (RAG) application, the performance of the vector database directly determines whether the large language model can quickly and accurately obtain external knowledge, thereby generating high-quality and timely content. The RAG system requires the vector database to achieve low-latency and high-throughput queries in massive (hundred million or even billion) data within milliseconds, which poses a great challenge to existing technologies.

[0004] Currently, the mainstream vector database technology solutions mainly fall into two categories: (1) monolithic / centralized databases, such as FAISS, HNSWlib index libraries deployed on a single or a few high-performance servers. This type of system performs excellently when the data size is small, but as the data volume explodes, its inherent "scale-up" mode soon encounters performance bottlenecks; (2) distributed databases. To solve the single-point performance problem, the industry has proposed a distributed solution that sharding and storing data on multiple servers to achieve "scale-out", and common data division strategies include random / hash division and clustering-based division.

[0005] Although existing vector database technologies have promoted the development of vector retrieval to some extent, when facing large-scale and high-concurrency retrieval requirements, especially in RAG and other scenarios with extremely high real-time requirements, there are still the following deficiencies: (1) The performance and cost of the centralized architecture are bottlenecks. The retrieval performance of a single database is completely limited by the hardware capabilities (CPU, memory, I / O) of a single server. When the scale of vector data reaches billions, the full index often cannot be fully loaded into the memory. Frequent disk I / O will sharply increase the query latency. Even if the performance is improved by upgrading the hardware, the cost will increase exponentially. There is also a physical limit to the performance improvement, which cannot meet the retrieval needs of larger-scale data in the future. (2) With the deepening of digital transformation in various industries, small and medium-sized enterprises and institutions are facing the need for efficient management and semantic retrieval of massive data (especially document data). However, many internal documents usually contain sensitive information and cannot rely on the vector databases provided by cloud service providers. Although locally deployed distributed vector databases (such as Milvus, Qdrant, etc.) can meet large-scale retrieval needs, their high requirements for computing resources (such as multi-core CPUs, large memory and high-speed storage) significantly increase hardware costs, making it difficult to adapt to the financial resources of small and medium-sized enterprises and institutions. Current technology makes it difficult to build a cluster with efficient retrieval capabilities based on old office computers with various chips. (3) Traditional distributed solutions have the problems of "pseudo-parallelism" and poor performance. Random / hash partitioning leads to low query efficiency. This strategy randomly scatters data to each node. Since the distribution of data has nothing to do with the similarity of the content, the relevant vectors of a query request may be scattered on any node. In order to ensure the recall rate (Recall), the query request must be broadcast to all data nodes, and then aggregate and sort after all nodes return the results. This "scatter-gather" mode will generate huge network overhead and communication delay when the number of nodes increases, and it does not achieve true query decoupling and efficient parallelism. The performance is extremely poor; clustering-based partitioning leads to representativeness bias and hotspot issues. This strategy (such as using the K-Means algorithm) groups vectors with similar content into the same shard. Although this can route queries to a few "relevant" shards in some scenarios, it has a fundamental flaw: each shard only represents a portion of the overall data, losing global representativeness. Therefore, it is impossible to obtain globally approximate results by "checking a point from every shard." In addition, this method is prone to "data hotspots" and "query hotspots." In other words, certain shards containing popular information will be frequently accessed, causing uneven load and becoming a system bottleneck again. (4) Existing solutions have high deployment and operation and maintenance costs. Whether it is through purchasing expensive high-performance servers for vertical expansion or building traditional distributed clusters that require complex load balancing and query routing strategies, existing solutions have extremely high requirements for resources. This not only brings high hardware procurement costs, but also makes many enterprises that want to use the idle resources of a large number of ordinary computers in the existing network discouraged. The threshold for technical deployment is too high. Summary of the Invention

[0006] In order to solve the problems of poor data partitioning strategy, high query latency, poor scalability and high deployment cost under the distributed architecture of existing vector databases, the present invention provides a distributed database system and data partitioning method based on low-discrepancy sequences, which scientifically and effectively partitions massive high-dimensional vector data, so that each data shard in distributed storage can maintain a high statistical representativeness of the overall data set, thereby overcoming the fundamental defects in query efficiency and effect brought about by traditional random partitioning or cluster partitioning; without modifying the native code of the lightweight stand-alone vector database and without increasing the cost of purchasing new hardware, various types of existing office computers are used to form a heterogeneous computing cluster, and the retrieval load is dispersed to the heterogeneous computing cluster composed of heterogeneous hardware through the proposed distributed task scheduling and index sharding strategy, thereby realizing high-performance distributed database retrieval.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a distributed database system based on low-discrepancy sequences.

[0008] A distributed database system based on low-discrepancy sequences includes: a coordination node and a plurality of data nodes communicating with the coordination node; The coordination node is configured to: receive data write requests and query requests from users or applications, perform data representative partitioning based on low-discrepancy sequences on the written data to obtain multiple data shards, and distribute the data shards to various data nodes for storage and local index construction; broadcast query tasks to all data nodes based on received query requests, collect local search results returned by all data nodes, perform global aggregation and sorting, and then return them to the user or application; The data node is configured to: receive and persistently store the data shards allocated by the coordination node, build local vector indexes for the persistently stored data shards, respond to query tasks from the coordination node, perform local retrieval and send the retrieval results to the coordination node.

[0009] In an implementation of the first aspect of the present invention, the coordination node includes at least: a data representativeness partitioning module; the data representativeness partitioning module includes: a vector ranking value generation submodule and a low-difference sequence grouping submodule; The vector sort value generation submodule is configured to: map any input high-dimensional vector into a one-dimensional sortable scalar value, thereby obtaining a vector data set sorted in ascending order according to the scalar value; The low-discrepancy sequence grouping submodule is configured to: generate a low-discrepancy sequence of the same length based on the total number of vectors of the written data, sort the low-discrepancy sequence, obtain the index information after the sequence is sorted, and rearrange and cut the vector data set that is sorted in ascending order according to the scalar value based on the index information to form multiple data fragments.

[0010] In an implementation of the first aspect of the present invention, the coordination node further includes: a data receiving and preprocessing module and a data distribution and database building control module; The data receiving and preprocessing module is configured to: receive data writing requests and query requests from users or applications, and perform preprocessing operations on the written data; The data distribution and database construction control module is configured to distribute data shards to designated data nodes and, after the distribution is completed, send instructions to the corresponding data nodes to enable the data nodes to persistently store the received data shards and build local vector indexes.

[0011] In an implementation of the first aspect of the present invention, the coordination node further includes: a parallel search control module and a result aggregation and sorting module; The parallel search control module is configured to: calculate the number of local queries that each data node needs to execute based on the received query vector and the total number of expected results K, and then broadcast the query vector and the number of local queries to all data nodes to start a parallel distributed search; The result aggregation and sorting module is configured to: collect local search results returned by all data nodes, generate a temporary list of candidate results, sort the candidate results in the temporary list according to the similarity score or similarity distance between the candidate results and the query vector, select the top K results from the sorted list as the final results, and return them to the user or application.

[0012] In an implementation of the first aspect of the present invention, a data node includes: a data storage and indexing module and a local search execution module; The data storage and indexing module is configured to: store data shards persistently on local disk or in memory, and build local vector indexes for the persistently stored data shards; The local search execution module is configured to: after receiving the query vector and the number of local queries n, use the local vector index to retrieve the n vectors most similar to the query vector and the corresponding identification codes and similarity scores or similarity distances, package the identification codes of the n vectors and the similarity scores or similarity distances with the query vector into a local result list, and send the local result list to the coordination node.

[0013] In an optional implementation of the first aspect of the present invention, a decentralized architecture is adopted, and the coordinating node is one of multiple data nodes. Any data node can be converted into a coordinating node and perform the functions of the coordinating node.

[0014] In an optional implementation of the first aspect of the present invention, the coordination nodes adopt various types of office computers, and the various types of office computers constitute a heterogeneous computing cluster with multiple coordination nodes.

[0015] In an optional implementation of the first aspect of the present invention, a coordination node cluster is constructed, which includes multiple coordination nodes. Each coordination node is connected through a consensus protocol communication to achieve master-slave switching of multiple coordination nodes.

[0016] In a second aspect, the present invention provides a data partitioning method based on low-difference sequences.

[0017] A data partitioning method based on low-discrepancy sequences includes the following steps: Get the original data set containing N high-dimensional vectors; Generate a one-dimensional sort value for each high-dimensional vector in the original data set D, and sort the original data set D in ascending order according to the one-dimensional sort value to obtain the data set D'; Based on the total number of vectors N in the original data set D, a one-dimensional low-discrepancy sequence X of length N is generated. The one-dimensional low-discrepancy sequence X is sorted, and the index of the original position of each element before sorting is recorded. Finally, a sorting index R of length N is obtained. According to the generated sorting index R, the ascending sorted data set D' is rearranged to obtain a new data set D''; When the number of data nodes is K, set the size of each data shard to M=N / K, and split the data set D'' into K data shards of size M in sequence; The generated K data shards are sent to K corresponding data nodes or a cluster consisting of multiple independent vector database instances that only provide basic retrieval services.

[0018] In an implementation of the second aspect of the present invention, generating a one-dimensional ranking value for each high-dimensional vector in the original data set D includes: A one-dimensional ranking value is obtained by principal component analysis; or a one-dimensional ranking value is obtained by a manifold learning algorithm; or an autoencoder is used to obtain a one-dimensional ranking value; or a feature weighting method is used to obtain a ranking value; or a value of a dimension that best represents the data distribution characteristics is directly selected as the one-dimensional ranking value; The Hua Luogeng-Wang Yuan method based on irrational numbers is used to generate low-discrepancy sequences; alternatively, the Halton sequence is used as the low-discrepancy sequence; alternatively, the Sobol sequence is used as the low-discrepancy sequence; alternatively, the Faure sequence is used as the low-discrepancy sequence.

[0019] In a third aspect, the present invention provides a distributed parallel retrieval method.

[0020] A distributed parallel search method, utilizing the distributed database system based on low-discrepancy sequences described in the first aspect of the present invention, comprises the following steps: The coordination node receives a query request from a user or an application, wherein the query request includes a query vector and a total number of results expected to be returned; The coordinating node determines the number of data nodes K participating in this query, calculates the number of local search results n that each data node needs to return, and broadcasts the query vector Q and the calculated number of local search results n to all data nodes. After receiving the query vector and the number of local queries n, the data node uses the local vector index to retrieve the most similar vectors and their corresponding identification codes and similarity scores or similarity distances, packing the identification codes of the n vectors and their similarity scores or similarity distances with the query vector into a local result list, and sending the local result list to the coordination node; The coordination node reorders all candidate vectors based on all received local result lists and selects the top K vectors as the final retrieval results.

[0021] In an implementation of the third aspect of the present invention, an approximate nearest neighbor algorithm is used to retrieve the vector most similar to the query vector. vectors, the approximate nearest neighbor algorithm is: a quantization-based method; or a tree-based method; or a graph-based method; or a hash-based method.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The application provides an economically feasible large-scale vector database deployment scheme, improves resource utilization of the system and reduces deployment cost, and by scientifically splitting mass data into multiple small-scale and high-representative shards, the whole system can be run on a cluster composed of general-purpose computers with ordinary performance, small hard disks and low memory, effectively avoiding dependence on expensive high-performance servers, greatly reducing hardware procurement and system maintenance cost, the database architecture of the application can fully utilize a large number of existing idle computing devices in enterprise networks, realizes effective utilization of resources, and clears the cost obstacle for popularization and application of large-scale vector retrieval technology.

[0023] The application provides an economically feasible large-scale vector database deployment scheme, by innovative architecture design, the office computers of medium and small enterprises and institutions can be fully utilized to form a heterogeneous computing cluster, bridge the gap between resource efficiency and scalability, without modifying the original code of the lightweight single-machine vector database and without increasing the hardware procurement cost, the existing various office computers (including various computers based on domestic chips) are used to form a heterogeneous computing cluster, by the innovative distributed task scheduling and index shard strategy, the retrieval load is dispersed to the office computer cluster composed of heterogeneous hardware, and high-performance distributed database retrieval is realized.

[0024] The application splits data by a low-discrepancy sequence (LDS) based on number theory, ensures that each data shard is a high-quality "miniature" of the overall data, realizes effective decomposition and parallelization of the computing task, and based on data splitting, a global query task originally unable to be executed on a single ordinary device due to excessive computing amount and memory demand can be successfully decomposed into multiple independent sub-query tasks executed in parallel on all shards, and the database architecture fundamentally solves the single-point computing bottleneck problem of the centralized system.

[0025] Although the vector database of the application runs on low-configuration hardware, since each data shard is highly representative of the whole, only a small number of most relevant results need to be retrieved from each shard for aggregation, and the Top-K result highly consistent with the global retrieval result on the high-performance server can be obtained, the low-cost deployment ensures high-quality retrieval result, ensures that the economy of the scheme is not at the expense of the retrieval quality, and can meet the strict requirements of RAG and other application scenarios on result accuracy.

[0026] Advantages of additional aspects of the application will be partially given in the following description, partially will become obvious from the following description, or will be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0028] Figure 1 A schematic diagram of the architecture of a distributed database system based on low-discrepancy sequences provided by an exemplary embodiment of the present invention; Figure 2 A schematic diagram of functional modules of a distributed database system based on low-discrepancy sequences provided by an exemplary embodiment of the present invention; Figure 3 A schematic flow chart of a data partitioning method based on low-difference sequences provided by an exemplary embodiment of the present invention; Figure 4 An exemplary embodiment of the present invention provides Figure 3 Detailed flow chart of step S2 in FIG. Figure 5 An exemplary embodiment of the present invention provides Figure 3 Detailed flow chart of step S4 in FIG. Figure 6 A schematic flow chart of a distributed parallel search method provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0031] As described in the background technology, there is a fundamental contradiction in the existing technology in how to achieve efficient, low-cost, and scalable retrieval of large-scale vector data. In view of this, the present implementation method proposes a new vector data partitioning and organization method and a distributed retrieval system based on this method, aiming to break the limitations of traditional solutions in hardware deployment and cost-effectiveness. Specifically, the present invention scientifically splits a vector database that is too large to run on a single ordinary server into multiple data shards, and ensures that each shard is a highly representative miniature of the overall data set in a statistical sense. In this way, the entire system can be deployed on a distributed cluster composed of low-cost, low-configuration general-purpose hardware, thereby fundamentally solving the problems of high hardware costs and high deployment thresholds of existing technical solutions while maintaining high-quality retrieval results, and can make full use of existing idle computing resources in the network.

[0032] At the system level, the present invention provides a distributed database system based on low-discrepancy sequences. The above-mentioned distributed vector database consists of one or more coordination nodes and multiple data nodes, which are interconnected through a network. The coordination nodes are responsible for managing metadata, executing data partitioning tasks and coordinating retrieval processes; the data nodes are responsible for storing data shards and performing local retrieval.

[0033] At the method level, the present invention includes two core processes: data representativeness partitioning and database building process and distributed parallel retrieval process.

[0034] The representative data partitioning and database construction process of this invention abandons traditional random partitioning or clustering strategies based on content similarity. Instead, it creatively adopts low-dissimilarity sequences based on number theory and statistics as the grouping basis. Through this process, massive vector datasets are divided into multiple data slices, ensuring that each slice is a highly representative sampling of the entire dataset in a statistical sense, achieving the effect of "minimizing inter-group differences and maximizing intra-group differences."

[0035] Based on the above unique division results, the distributed parallel retrieval process of the present invention can efficiently decompose the global retrieval task to all data nodes for parallel execution. Since each data shard is approximately a "miniature" of the overall data, it is only necessary to retrieve a small number of the most relevant results from each node for aggregation to obtain a high-quality approximation of the final result of the global retrieval.

[0036] In summary, the present invention uses a new data organization method to construct a distributed vector database system that truly supports efficient parallel retrieval in its architecture, thereby realizing the ability to perform large-scale, high-precision vector retrieval on low-cost, general-purpose hardware and having excellent scalability.

[0037] like Figure 1 FIG. 1 is a schematic diagram of a distributed database system based on low-difference sequences provided by the present invention, which includes three core components: a coordination node, a data node, and a network connecting them.

[0038] The coordination node is the "brain" and main entrance of the entire distributed system. It can be a standalone server or a high-availability cluster consisting of multiple servers. Its main responsibilities include: Request processing: Serves as the unified interface of the system, receiving data write requests and query requests from users or applications; Data partitioning: Execute the core technology of this invention, namely "data representative partitioning based on low-discrepancy sequences", to divide the received massive vector data into multiple data slices; Metadata management: Maintains metadata information for the entire cluster, including the address and status of data nodes, data sharding information stored on each node, and summary information of global indexes; Task distribution and coordination: Distribute the divided data shards to each data node for storage and local index construction. When processing query requests, broadcast the query tasks to all (or specified) data nodes and coordinate the parallel retrieval process. Result aggregation: Collect local search results returned by all data nodes, perform global aggregation and sorting, and return the final results to the user.

[0039] A data node is a unit in a distributed system responsible for data storage and computation execution. A system typically consists of multiple data nodes forming a cluster. The size of the cluster can be dynamically scaled based on the amount of data and query load. Its main responsibilities include: Data storage: Receives and persistently stores data shards assigned by the coordinating node. Each data node is typically responsible for one or more shards. Local index construction: The stored data is sharded and local vector indexes are constructed using efficient approximate nearest neighbor (ANN) algorithms such as HNSW and IVF to accelerate local queries. Local retrieval: In response to query instructions from the coordinating node, it performs vector similarity search on the local index and returns a specified number of local results that are most similar to the query vector; Node Scalability: The number of data nodes can be expanded horizontally as needed. New nodes can share storage and query workloads, linearly increasing the capacity and performance of the entire system. A key advantage of this invention is that its solution has low hardware requirements for data nodes. These data nodes can be dedicated servers or ordinary computers with idle computing resources within the enterprise network, effectively utilizing existing hardware resources and achieving low-cost deployment.

[0040] The network is the channel that connects the coordination node with all data nodes, and for communication between nodes. It is usually a standard TCP / IP Ethernet. The bandwidth and latency of the network are one of the key factors affecting the overall performance of the system, especially in the data distribution and query result aggregation stages.

[0041] To implement the technical solution of the present invention, the vector database system of the present invention is logically divided into a series of functional modules. These modules are distributed on the coordination nodes and data nodes and work together to complete data partitioning, database construction and retrieval. Figure 2 shown.

[0042] The functional modules mainly include: data reception and preprocessing module on the coordination node, data representativeness division module, data distribution and database construction control module, parallel retrieval control module, result aggregation and sorting module, as well as data storage and indexing module and local retrieval execution module on the data node.

[0043] The data reception and preprocessing module serves as the entry point for data inflow, responsible for receiving batches of high-dimensional vector datasets from external users or systems. This module performs preliminary preprocessing operations on the data, such as format verification, cleaning, and normalization, to prepare standardized data for the subsequent segmentation steps.

[0044] The data representativeness partitioning module is responsible for executing an innovative data partitioning algorithm to divide the overall data set into multiple data slices with high statistical representativeness. It consists of a vector ranking value generation submodule and a low difference sequence (BDS) grouping submodule.

[0045] The vector sorting value generation submodule is responsible for mapping the input high-dimensional vector into a one-dimensional, sortable scalar value that reflects the distribution characteristics of the vector in a certain main dimension.

[0046] In a preferred embodiment of the present invention, the module uses the principal component analysis (PCA) method to calculate the first principal component of the data, and uses the projection value of each vector on the principal component as its one-dimensional ranking value.

[0047] Optionally, in some other implementations, any method that can generate a stable and discriminative one-dimensional scalar value for a high-dimensional vector falls within the scope of protection of the present invention, such as using a manifold learning algorithm such as UMAP, t-SNE to reduce it to a one-dimensional output, or other feature selection and projection functions.

[0048] The low-discrepancy sequence (BDS) grouping submodule is responsible for generating the results of the submodule based on the vector sorting value, and combining it with the low-discrepancy sequence theory to group all the data. Specifically, it includes: first, generating a low-discrepancy sequence of corresponding length based on the total number of vectors N in the data set; then, obtaining the index information of the sorted sequence; finally, rearranging and cutting the vector data set that has been sorted by the sorting value based on this index information, and finally forming K data shards. This process ensures that each shard is a uniform and unbiased sampling of the overall data distribution.

[0049] The data distribution and database construction control module is responsible for distributing the data shards generated by the "Data Representative Partitioning Module" to designated data nodes through the network. After the distribution is completed, the module will send instructions to the corresponding data nodes, requiring them to persist the received data shards and build local vector indexes.

[0050] The parallel retrieval control module is responsible for processing user query requests. When it receives a query vector Q and the total number of expected results K, the module calculates the number of results n that each data node should return for the local query (for example, n is equal to the ratio of K to the number of nodes). It then broadcasts the query vector Q and parameter n to all data nodes in the cluster, thereby initiating a parallel distributed retrieval.

[0051] The result aggregation and ranking module is responsible for collecting the local search results returned by all data nodes. Since each node returns n most similar results, the module will get a list containing “n Then, the result aggregation and sorting module will perform a global and accurate sorting based on the original similarity scores (or distances) between these candidate results and the query vector, and select the top K from the sorted list as the final results and return them to the user.

[0052] The data storage and indexing module is responsible for receiving, managing, and indexing data shards assigned by the coordinating node. This module persistently stores data shards on local disk or in memory and builds local index files for these data using efficient approximate nearest neighbor (ANN) search algorithms (such as HNSW, IVF-PQ, and LSH). This index targets only local data shards, is small in size, and is fast to build and load.

[0053] The local retrieval execution module is responsible for executing specific retrieval tasks from the coordination node. After receiving the query vector Q and the number of local queries n, the module will use the local index built by the "data storage and indexing module" to quickly retrieve the n vectors most similar to Q and the corresponding IDs (i.e., identification codes) and similarity scores, and return this result list to the coordination node through the network.

[0054] It should be noted that the above solution of the present invention describes a centralized coordination architecture consisting of coordination nodes and data nodes. However, the core partitioning and retrieval ideas of the present invention are also applicable to other distributed architectures.

[0055] For example, a high-availability coordination node architecture can be optionally adopted. To avoid single point failure of the coordination node, a master-slave mode can be adopted, or by introducing components such as ZooKeeper and etcd, a coordination node cluster can be built based on the Raft / Paxos consensus protocol to achieve high availability.

[0056] Another example is the adoption of a decentralized / peer-to-peer architecture. In certain application scenarios, the centralized role of the coordinating node can be weakened. Each node acts as a data node to store data shards and also coordinates metadata synchronization and query routing through peer-to-peer network protocols such as Gossip. Query requests can be sent to any node in the network, which then initiates broadcasts and aggregates results.

[0057] Alternatively, optionally, a data partitioning and retrieval service separation architecture can be adopted, and the data partitioning function of the present invention can be implemented as an independent, offline "data preprocessing service", which is responsible for regularly receiving new data, performing representative partitioning, and pushing the generated data shards to a cluster consisting of multiple independent vector database instances that only provide basic retrieval services.

[0058] The database system of the present invention can be deployed not only on a server cluster, but also flexibly deployed in various modern computing environments, such as: It can be deployed on a public or private cloud using virtual machines or containers, and automated management, scaling, and fault recovery can be achieved through container orchestration tools. Alternatively, the coordination node (or high-availability cluster) can be deployed on a central server with better performance, while the data nodes can be widely deployed on a large number of ordinary performance computers, workstations, and even edge computing devices in the enterprise network to form a heterogeneous hybrid computing cluster.

[0059] Based on the above-mentioned distributed vector database system based on low-discrepancy sequences, this implementation proposes a data partitioning method based on low-discrepancy sequences. This data partitioning method describes how the present invention processes a large-scale original vector data set into multiple data slices with high statistical representativeness through an innovative partitioning method, and finally completes the index construction process on each data node of the distributed system. This process is mainly scheduled and executed by the coordination node. Specifically, Figure 3 As shown, the following process is included: Step S1: Receive the original data set.

[0060] At the beginning of the process, the data receiving and preprocessing module of the coordination node receives a raw data set containing N high-dimensional vectors ,In this step, the vector can be preprocessed such as normalization to ensure the consistency of ,the data.

[0061] Step S2: Generate a one-dimensional sort value and perform preliminary sorting on the data.

[0062] like Figure 4As shown, in order to be able to group based on one-dimensional sequences later, the vector ranking value generation submodule in the data representativeness partitioning module of the coordination node generates each high-dimensional vector in the data set D. Process it and generate a corresponding one-dimensional scalar sort value for it .

[0063] One possible approach is to use principal component analysis (PCA) to first calculate the first principal component axis of the entire data set D, and then transform each vector Projected onto the axis, the projection value is the one-dimensional sort value of the vector ; After the processing is completed, a one-dimensional sorted value sequence corresponding to the vectors in D is obtained ; Then, the original data set D is sorted in ascending (or descending) order according to the value of sequence E to obtain a preliminary sorted data set ,in is sorted by its value The permuted vector.

[0064] Optionally, in some other implementations, other dimensionality reduction methods can also be used. For example, algorithms such as UMAP (Uniform Manifold Approximation and Projection) and t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used to reduce the dimension to one dimension as the ranking value; alternatively, an autoencoder model can be trained and the one-dimensional output of its bottleneck layer (Bottleneck Layer) can be used as the ranking value of each vector; alternatively, for vectors in certain specific fields, a weighted summation can be performed on specific dimensions in the vector based on prior knowledge, or a dimension that best represents the data distribution characteristics can be directly selected as the ranking value.

[0065] Step S3: Generate low-discrepancy sequences (BDS) and their sorting indexes.

[0066] The low-discrepancy sequence (BDS) grouping submodule in the data representative partitioning module of the coordination node generates a one-dimensional low-discrepancy sequence of length N based on the total number of vectors N in the data set. .

[0067] A low-variance sequence is a deterministic sequence that is more evenly distributed in space than a pseudo-random sequence. In one embodiment of the present invention, the Hua Luogeng-Wang Yuan method can be used to generate the sequence. For example, an irrational number α (such as π) is taken, and the i-th element of the sequence is can be Calculated.

[0068] It should be noted that other deterministic or quasi-random sequence generation algorithms with good uniform distribution properties are also applicable, for example: Halton Sequence: uses coprime integers as bases to generate sequences; Sobol' Sequence: Another widely used low-discrepancy sequence; Faure Sequence: Constructed based on the van der Koppiet sequence under a specific cardinality.

[0069] After generating the sequence X, sort it and record the index of each element's original position before sorting. Finally, a sort index (Permutation Index) of length N is obtained. ,in Indicates that the element at position i after sorting comes from the original sequence X positions, this index R is the key to achieving representative grouping.

[0070] Step S4: Perform representative grouping according to the sorting index.

[0071] like Figure 5 As shown, the coordination node uses the sorting index R generated in step S3 to rearrange and group the dataset D' obtained in step S2 after preliminary sorting. Specifically, a new empty dataset D'' is created, and vectors are taken from D' and put into D'' in the order of index R. That is, the i-th vector of D'' is the i-th vector in D'. This rearrangement operation breaks the local clustering that may occur in step S2 due to content similarity (similar projection values ​​on the principal components). Through the uniform distribution characteristics of BDS, vectors with different characteristics are evenly distributed in the entire D'' sequence.

[0072] Assuming the number of data nodes in the system is K, the size of each data shard is M=N / K, and the rearranged data set D'' is divided into K data shards of size M in sequence. , , ..., .Right now , ,..., .

[0073] After this step, each data shard obtained The vectors inside have a large difference (from different regions), while different shards and The overall data distribution between them is highly similar, and each shard constitutes a high-quality, unbiased representative sample of the original dataset D.

[0074] Step S5: Distribute data shards.

[0075] The data distribution and database construction control module of the coordination node starts the data distribution task and divides the K data fragments generated in step S4 into , ,..., , and sent to K corresponding data nodes through the network.

[0076] Step S6: Build local indexes in parallel.

[0077] After each data node receives the data shard assigned to it, its data storage and indexing module starts working. This module stores the data shards persistently and independently and in parallel uses ANN algorithms such as HNSW to build vector indexes for local data. This step has two advantages: fast construction speed. Since the amount of data processed by each node is only 1 / K of the total amount, the index construction process is very fast; low memory usage, that is, the peak memory consumption required for index construction is significantly reduced. This solves the memory overflow problem caused by building indexes for massive data at one time under a monolithic architecture, allowing the entire database construction process to be successfully executed on low-configuration devices with limited memory resources. After all nodes complete index construction, the database construction process of the entire distributed vector database is completed, and the system enters a queryable state.

[0078] Based on the above-mentioned distributed vector database system based on low-discrepancy sequences, this implementation proposes a distributed parallel retrieval method. It describes how, when the distributed vector database system receives a user's query request, each node works together to efficiently complete the query task on the distributed architecture and accurately return the most similar vector results. The core of this process is to utilize the high representativeness of each data shard to the overall data, successfully decomposing a large-scale global search task that could not be executed on a single low-configuration device into multiple small-scale local search tasks that can be processed in parallel on each node. Specifically, Figure 6 As shown, the following steps are included: Step T1: Receive a query request.

[0079] The process begins when the parallel retrieval control module of the coordination node receives a query request from a user or application. The request usually contains two main parameters: Query vector Q: a high-dimensional vector with the same dimension as the vectors stored in the database; The total number of results K expected to be returned is the number of most similar vectors that the user wants to obtain.

[0080] Step T2: Calculate local search number and broadcast query.

[0081] The parallel search control module of the coordination node performs the following operations according to the query request and the current system state: Determine the number of active data nodes K_active participating in this query. In the normal case, K_active equals the total number of data nodes K of the system to maximize the utilization of parallel capabilities.

[0082] Calculate the number of local search results n that each data node needs to return. A preferred calculation method is n = ceil (K / K_active), where ceil is the ceiling function. This is done to ensure that the total number of candidate results collected is at least K even in the worst case. In practice, in order to improve the accuracy of the final results, n can also be set to a value slightly larger than K / K_active, for example where α is a redundancy factor greater than 1 (such as 1.1 or 1.2).

[0083] Broadcast the query vector Q and the calculated local search number n to all K_active active data nodes through the network.

[0084] Step T3: Each data node performs local search in parallel.

[0085] After each data node in the cluster receives the query instruction from the coordination node, its local search execution module immediately, independently and in parallel performs the following operations: Perform an approximate nearest neighbor (ANN) search on the vector index corresponding to the locally stored data shard.

[0086] Find the n vectors with the highest similarity (or closest distance) to the query vector Q.

[0087] Pack the IDs of these n vectors and the similarity scores (or distances) to the query vector Q into a local result list.

[0088] It should be noted that the present application does not have specific restrictions on the approximate nearest neighbor (ANN) search algorithm used locally by the data nodes, and any efficient ANN index algorithm can be used as an alternative solution, including but not limited to: Quantization-based methods: such as IVF-PQ (Inverted File with Product Quantization); Tree-based methods: such as Annoy (Approximate Nearest Neighbors Oh Yeah); Graph-based methods (preferred): such as HNSW (Hierarchical Navigable Small World), are used in the preferred embodiment; Hash-based methods: such as LSH (Locality-Sensitive Hashing).

[0089] Step T4: Collect all local results.

[0090] After each data node completes its local search, it sends its own local result list back to the coordination node via the network. The result aggregation and sorting module of the coordination node is responsible for monitoring and receiving the results returned by all data nodes.

[0091] Step T5: Global aggregation and reordering.

[0092] After receiving the return results from all data nodes (or within the set timeout period), the result aggregation and sorting module of the coordination node performs the following operations: List all local results (total candidates) into a single global candidate list.

[0093] All vectors in this global candidate list are globally and accurately re-ranked according to their similarity scores (or distances) with the query vector Q. This step eliminates ranking biases that may be caused by differences in local data distribution among nodes, ensuring the accuracy of the final result.

[0094] Step T6: Select the global Top-K and return.

[0095] From the global candidate result list sorted in step T5, the top K ranked vectors are selected as the final search results. The coordinating node returns this final result list to the requesting user or application. This completes a complete distributed parallel search process.

[0096] The distributed vector database system, data partitioning method, and retrieval method based on low-discrepancy sequences provided by the present invention have the following significant and fundamental beneficial effects in solving the practical deployment challenges of large-scale vector retrieval: (1) Greatly reduces the hardware threshold and deployment costs, and realizes resource reuse.

[0097] The core advantage of this invention is that it makes it possible to deploy and run large-scale vector databases on a cluster composed of low-cost, low-configuration general-purpose hardware. Specifically, it includes: Solve single-point hardware bottlenecks: Conventional monolithic databases, when faced with massive amounts of data, place extremely high demands on the memory, CPU, and storage performance of a single server, often costing hundreds of thousands or even millions of yuan. This invention uses scientific data partitioning to decompose a large-scale database, which, for example, requires hundreds of GB of memory to load indexes, into multiple small database shards that only require a few GB of memory. This fundamentally solves the problem of deployment problems due to hardware limitations. Effective utilization of idle resources is achieved: The present invention has extremely low hardware configuration requirements for data nodes, so that it can be flexibly deployed on a large number of existing ordinary servers or personal computers with idle computing and storage capabilities in the enterprise network; this revitalizes sunken IT assets, achieves effective utilization of resources, avoids large-scale hardware procurement for a single application scenario, and is extremely cost-effective.

[0098] (2) Under low-cost deployment, the retrieval accuracy is highly consistent with the centralized solution.

[0099] While achieving low-cost deployment, the present invention does not sacrifice search quality. Its unique representative partitioning method ensures high accuracy of search results. Specifically, it includes: High-quality global result approximation: Partitioning based on low-discrepancy sequences ensures that each data shard is a statistically high-quality "microcosm" of the entire dataset. Therefore, the local optimal results retrieved from each shard can be aggregated to form a high-quality candidate set that covers the global data distribution, thus ensuring the accuracy of the final result. Experimental data support: In a comparative experiment conducted on a test dataset, the following indicators were used to measure the quality of retrieval results: Match Rate: refers to the proportion of results whose categories match the original category of the query text among the K results returned. This metric measures the macro-relevance of the search results and is calculated as follows: Match rate = (number of category matches in the returned results / total number of returned results K) × 100%; First Result Distance: This refers to the distance between the result that is most similar to the query vector (i.e., the smallest distance) among the K results returned. This metric measures the system's ability to find the best match. A smaller distance indicates a higher degree of match. Average Distance: This refers to the arithmetic average of the distances between the K results returned and the query vector. This metric reflects the overall similarity level of the returned result set. When retrieving the top 10 results, the experimental results show that: The distributed search solution of the present invention achieved an average match rate of 50.50%, while the traditional centralized search solution running on a single high-performance server achieved an average match rate of 51.21%, a difference of less than 1 percentage point. In terms of the mean of the first result distance, the two are completely consistent. In terms of the mean of the average distance of each result, the mean distance of the method using the present invention is 0.3025, and the mean distance of the centralized query is 0.2876, which is also very small. Experimental data strongly proves that while the present invention reduces hardware costs by several orders of magnitude, the accuracy of its retrieval results is almost the same as that of expensive centralized solutions, successfully achieving the goal of "low cost and high quality".

[0100] (3) This invention realizes the effective decomposition of computing tasks and provides a solid foundation for system scalability.

[0101] This invention provides a perfect architectural foundation for horizontal expansion of the system by decomposing a large computing task into multiple parallel subtasks. Specifically, it includes: Decomposing Computational and Storage Pressures: A global search or index building task, which would have been prohibitive on a single standard device due to its computational and memory requirements, is now successfully decomposed into multiple independent subtasks that execute in parallel across all shards. This not only resolves the "can it run" issue but also means that as data volume or concurrent requests increase, the system's capacity can be linearly scaled by simply adding standard hardware nodes. Reduces single-point performance bottlenecks: Although the end-to-end total time of a distributed solution may be higher than that of an idealized single-machine memory calculation due to network communication overhead, this invention eliminates the reliance on single-point high-performance computing. In many business scenarios, "being able to complete a task within 1 minute with a low-cost solution" is far more practical and feasible than "having to spend a huge amount of money to complete it within 10 seconds."

[0102] (4) This invention promotes the popularization of large-scale, high-precision AI applications.

[0103] The low-cost, high-quality features of the present invention greatly lower the deployment threshold of advanced AI applications such as Retrieval Augmentation Generation (RAG), making large-scale RAG applications possible. In existing solutions, building a vector knowledge base for the RAG system, which often requires TB-level memory and high-end CPU clusters, is very expensive, making many companies reluctant to do so. The present invention provides an economically feasible alternative, allowing companies to use existing IT resources to deploy large-scale knowledge bases that were previously unaffordable, achieving a qualitative change from "too costly to do" to "now economically doable", greatly promoting the popularization and application of technologies such as RAG.

[0104] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A distributed database system based on low-discrepancy sequences, It is characterized in that It includes: a coordination node and a plurality of data nodes communicating with the coordination node; The coordination node is configured to: receive data write requests and query requests from users or applications, perform data representative partitioning based on low-discrepancy sequences on the written data to obtain multiple data shards, and distribute the data shards to various data nodes for storage and local index construction; broadcast query tasks to all data nodes based on received query requests, collect local search results returned by all data nodes, perform global aggregation and sorting, and then return them to the user or application; The data node is configured to: receive and persistently store the data shards allocated by the coordination node, build local vector indexes for the persistently stored data shards, respond to query tasks from the coordination node, perform local retrieval and send the retrieval results to the coordination node.

2. The distributed database system based on low-discrepancy sequences according to claim 1, wherein: The coordination node at least includes: a data representativeness division module; the data representativeness division module includes: a vector ranking value generation submodule and a low difference sequence grouping submodule; The vector sort value generation submodule is configured to: map any input high-dimensional vector into a one-dimensional sortable scalar value, thereby obtaining a vector data set sorted in ascending order according to the scalar value; The low-discrepancy sequence grouping submodule is configured to: generate a low-discrepancy sequence of the same length based on the total number of vectors of the written data, sort the low-discrepancy sequence, obtain the index information after the sequence is sorted, and rearrange and cut the vector data set that is sorted in ascending order according to the scalar value based on the index information to form multiple data fragments.

3. The distributed database system based on low-discrepancy sequences according to claim 1 or 2, characterized in that: The coordination node also includes: a data receiving and preprocessing module and a data distribution and database building control module; The data receiving and preprocessing module is configured to: receive data writing requests and query requests from users or applications, and perform preprocessing operations on the written data; The data distribution and database construction control module is configured to distribute data shards to designated data nodes and, after the distribution is completed, send instructions to the corresponding data nodes to enable the data nodes to persistently store the received data shards and build local vector indexes.

4. The distributed database system based on low-discrepancy sequences according to claim 1 or 2, wherein: The coordination node further includes: a parallel search control module and a result aggregation and sorting module; The parallel search control module is configured to: calculate the number of local queries that each data node needs to execute based on the received query vector and the total number of expected results K, and then broadcast the query vector and the number of local queries to all data nodes to start a parallel distributed search; The result aggregation and sorting module is configured to: collect local search results returned by all data nodes, generate a temporary list of candidate results, sort the candidate results in the temporary list according to the similarity score or similarity distance between the candidate results and the query vector, select the top K results from the sorted list as the final results, and return them to the user or application.

5. The distributed database system based on low-discrepancy sequences according to claim 1 or 2, wherein: The data node includes: a data storage and indexing module and a local search execution module; The data storage and indexing module is configured to: store data shards persistently on local disk or in memory, and build local vector indexes for the persistently stored data shards; The local search execution module is configured to: after receiving the query vector and the number of local queries n, use the local vector index to retrieve the n vectors most similar to the query vector and the corresponding identification codes and similarity scores or similarity distances, package the identification codes of the n vectors and the similarity scores or similarity distances with the query vector into a local result list, and send the local result list to the coordination node.

6. The distributed database system based on low-discrepancy sequences according to claim 1, wherein: A decentralized architecture is adopted, where the coordinating node is one of multiple data nodes, and any data node can be converted into a coordinating node and perform the functions of the coordinating node; Alternatively, a coordination node cluster is constructed, which includes multiple coordination nodes. Each coordination node is connected through a consensus protocol to achieve master-slave switching of multiple coordination nodes. Alternatively, a coordination node cluster is constructed, where the coordination nodes are various office computers, and the coordination node cluster is composed of multiple coordination nodes composed of various office computers.

7. A data partitioning method based on low-difference sequences, characterized in that: The following processes are included: Get the original data set containing N high-dimensional vectors; Generate a one-dimensional sort value for each high-dimensional vector in the original data set D, and sort the original data set D in ascending order according to the one-dimensional sort value to obtain the data set D'; Based on the total number of vectors N in the original data set D, a one-dimensional low-discrepancy sequence X of length N is generated. The one-dimensional low-discrepancy sequence X is sorted, and the index of the original position of each element before sorting is recorded. Finally, a sorting index R of length N is obtained. According to the generated sorting index R, the ascending sorted data set D' is rearranged to obtain a new data set D''; When the number of data nodes is K, set the size of each data shard to M=N / K, and split the data set D'' into K data shards of size M in sequence; The generated K data shards are sent to K corresponding data nodes or a cluster consisting of multiple independent vector database instances that only provide basic retrieval services.

8. The data partitioning method based on low-difference sequences according to claim 7, wherein: Generate a one-dimensional ranking value for each high-dimensional vector in the original data set D, including: A one-dimensional ranking value is obtained by principal component analysis; or a one-dimensional ranking value is obtained by a manifold learning algorithm; or an autoencoder is used to obtain a one-dimensional ranking value; or a feature weighting method is used to obtain a ranking value; or a value of a dimension that best represents the data distribution characteristics is directly selected as the one-dimensional ranking value; The Hua Luogeng-Wang Yuan method based on irrational numbers is used to generate low-discrepancy sequences; alternatively, the Halton sequence is used as the low-discrepancy sequence; alternatively, the Sobol sequence is used as the low-discrepancy sequence; alternatively, the Faure sequence is used as the low-discrepancy sequence.

9. A distributed parallel search method, characterized in that: Utilizing the distributed database system based on low-discrepancy sequences according to any one of claims 1 to 6, comprising the following process: The coordination node receives a query request from a user or an application, wherein the query request includes a query vector and a total number of results expected to be returned; The coordinating node determines the number of data nodes K participating in this query, calculates the number of local search results n that each data node needs to return, and broadcasts the query vector Q and the calculated number of local search results n to all data nodes. After receiving the query vector and the number of local queries n, the data node uses the local vector index to retrieve the most similar vectors and their corresponding identification codes and similarity scores or similarity distances, packing the identification codes of the n vectors and their similarity scores or similarity distances with the query vector into a local result list, and sending the local result list to the coordination node; The coordination node reorders all candidate vectors based on all received local result lists and selects the top K vectors as the final retrieval results.

10. The distributed parallel search method according to claim 9, wherein: The approximate nearest neighbor algorithm is used to retrieve the vector most similar to the query vector. vectors, the approximate nearest neighbor algorithm is: quantization-based method; Alternatively, a tree-based method; or a graph-based method; or a hash-based method.

Images