Vector database construction method and system

Abstract

The invention provides a vector database construction method and system, and relates to the technical field of databases. The method comprises the steps of performing normalization processing on an original vector so as to limit a sudden change value and zoom the sudden change value to a unified range; recursive dichotomy is carried out on the value range of each dimension, binary codes are allocated to each interval, and mapping of the intervals and the codes is dynamically recorded through a skip list; alternately combining codes of all dimensions of the normalized vector to generate a global binary string; and finally, taking the binary string and the original vector as a joint primary key, and constructing a B + tree index. According to the method, a high-dimensional proximity problem is converted into a prefix matching problem of a one-dimensional space, and excellent characteristics of a B + tree and a skip list are utilized, so that efficient insertion, deletion, modification, accurate query and fuzzy query of high-dimensional vector data are realized; the problems of complicated calculation, high storage overhead, difficulty in dynamic updating and low query speed of the traditional method on high-dimensional data are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of database technology, and specifically to a method and system for constructing a vector database. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, vector databases have become core tools for processing high-dimensional vector data (such as embedded vectors in text, images, and audio). However, existing vector databases face significant challenges in their design and implementation: 1. High computational complexity: The time complexity of accurately calculating the distance between high-dimensional vectors (such as Euclidean distance) is proportional to the dimension, making it difficult to achieve real-time response on large-scale datasets.

[0003] 2. High storage overhead: High-dimensional data requires complex index structures, and traditional methods have high storage costs, making it difficult to support ultra-large-scale datasets.

[0004] 3. Difficulty in dynamic updates: Most high-performance index structures (such as graph-based HNSW) sacrifice dynamism in pursuit of query efficiency, making it difficult to support real-time and frequent data insertion and deletion operations, thus limiting their application in dynamic scenarios.

[0005] 4. Slow query speed: When dealing with massive amounts of data, whether it is a linear scan or a complex index traversal, the query latency is difficult to meet the needs of real-time applications.

[0006] Therefore, there is an urgent need in this field for a vector database construction scheme that can balance efficient querying, low storage overhead, and support dynamic updates. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for constructing a vector database to solve at least one technical problem existing in the prior art.

[0008] On one hand, embodiments of the present invention provide a method for constructing a vector database, the method comprising: step S1, restricting the values ​​of each dimension of the original vector within a preset range, and normalizing the values ​​of each dimension within the preset range based on a preset scaling factor to obtain a normalized vector; step S2, performing a recursive binary division on the preset range to obtain N sub-intervals; step S3, assigning corresponding binary codes to the N sub-intervals based on preset encoding rules; step S4, constructing a mapping table for each dimension to store the mapping relationship between each sub-interval and the binary code; step S5, for the sub-intervals to which each dimension value of the normalized vector belongs, querying the binary code corresponding to the sub-interval based on the mapping table, and interleaving the binary codes of all dimensions to generate a global binary string; step S6, using the global binary string and the original vector as a joint index key to construct a B+ tree index, based on the B+ tree index supporting insertion, deletion, modification, precise query, and fuzzy query operations on vector data.

[0009] Furthermore, step S1 includes: step S101, replacing the dimension values ​​in the original vector that exceed the upper limit value of the preset range with the upper limit value, and replacing the dimension values ​​that are lower than the lower limit value of the preset range with the lower limit value; step S102, scaling the vector after value restriction processing using a preset linear transformation formula so that all its dimension values ​​fall within a uniform normalization range.

[0010] Furthermore, the termination condition for the recursive binary division is reaching a preset division depth; the division depth determines the number of intervals and the encoding length of each dimension; the division depth is configured in a balanced manner based on the required precision and available storage resources.

[0011] Furthermore, the method also includes a precision control step: by adjusting the depth of the recursive binary search partitioning, the balance between system precision and storage overhead is controlled; when the partitioning depth increases, the spatial partitioning precision improves, and the corresponding mapping table size increases; when the partitioning depth decreases, the spatial partitioning precision decreases, and the corresponding mapping table size decreases.

[0012] Furthermore, the preset encoding rule is that during each binary division, a binary bit 0 is assigned to the left interval and a binary bit 1 is assigned to the right interval; the final binary code of a sub-interval is the sequential concatenation of all the assigned binary bits along the path from the initial interval to the current interval.

[0013] Furthermore, the mapping table is a skip list structure; the skip list dynamically records the mapping relationship between the range of the divided sub-intervals and the binary code through a multi-level index structure; the skip list supports interval query operations with logarithmic time complexity, and supports efficient insertion and deletion operations.

[0014] Furthermore, the interleaving combination method described in step S5 is as follows: if the vector has M dimensions, the encoding of each dimension is as follows: , ,......, Each code has a length of L. When combining, take the first bit of all dimension codes in sequence, arrange them in dimension order, then take the second bit of all dimension codes in sequence, and so on, until all L bits are taken. Concatenate all the binary bits obtained in this order to form the global binary string.

[0015] Furthermore, the fuzzy query operation specifically includes: calculating the global binary string of the target vector; performing a prefix matching query based on the binary string in the B+ tree index to locate the leaf node region storing similar codes; and using the chained pointers between the leaf nodes of the B+ tree, traversing the leaf node and its multiple adjacent leaf nodes to collect a set of candidate vectors.

[0016] Furthermore, the fuzzy query operation also includes a reordering step: after collecting the candidate vector set, calculating the precise distance between each candidate vector in the candidate vector set and the query target vector; sorting the candidate vectors according to the precise distance, and returning the K vectors with the smallest distance as the final query result.

[0017] Secondly, embodiments of the present invention provide a vector database construction system, which is implemented using the aforementioned vector database construction method. The system includes: a normalization module, adapted to restrict the values ​​of each dimension of the original vector within a preset range, and normalize the values ​​of each dimension within the preset range based on a preset scaling factor to obtain a normalized vector; a spatial partitioning and encoding management module, adapted to perform recursive binary division on the preset range to obtain N partitioned sub-intervals; assign corresponding binary codes to the N partitioned sub-intervals based on preset encoding rules; construct a mapping table for each dimension to store the mapping relationship between each partitioned sub-interval and the binary code; a vector encoding module, adapted to obtain the partitioned sub-intervals to which each dimension value belongs for the normalized vector, query the binary code corresponding to the partitioned sub-interval based on the mapping table, and interleave the binary codes of all dimensions to generate a global binary string; and a B+ tree storage engine module, adapted to use the global binary string and the original vector as a joint index key to construct a B+ tree index, which supports insertion, deletion, modification, precise query, and fuzzy query operations on vector data.

[0018] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the above-described vector database construction method.

[0019] Fourthly, embodiments of the present invention also provide a readable storage medium, wherein when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to execute the above-described vector database construction method.

[0020] The beneficial effects of this invention are as follows: 1. Efficient dynamic operations: By managing dynamic space partitioning through skip lists, it supports range queries and updates with O(log n) time complexity. Combined with the inherent dynamic balancing characteristics of B+ trees, the entire system can efficiently handle real-time data insertion and deletion.

[0021] 2. Fast Proximity Search: Utilizing the prefix matching property of alternating combination encoding, vectors that are geographically close in high-dimensional space have a long common prefix in their encoding, thus being clustered together in physical storage by the B+ tree index. During fuzzy queries, prefix matching and leaf node chain traversal can quickly narrow down the candidate set, avoiding high-dimensional distance calculations on the entire dataset and significantly improving query efficiency.

[0022] 3. Excellent scalability: It supports independent partitioning and encoding of multidimensional data, and can flexibly control the balance between accuracy and storage cost by adjusting the depth of recursive partitioning. It is suitable for various high-dimensional vector data scenarios ranging from small to very large scale.

[0023] 4. Storage structure optimization: Using compact binary strings as the primary key component, and combining them with the highly mature and optimized B+ tree disk storage structure, storage overhead is effectively reduced while ensuring system stability and reliability. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] Figure 1 This is a flowchart of a vector database construction method provided in Embodiment 1 of the present invention.

[0026] Figure 2 This is an illustrative diagram illustrating a high-dimensional space partitioning example provided in Embodiment 1 of the present invention.

[0027] Figure 3 This is a schematic diagram of a vector database construction system provided in Embodiment 2 of the present invention.

[0028] Figure 4This is a partial block diagram of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0029] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0030] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0031] The present invention will now be described in detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0032] Example 1 For ease of understanding, the overall inventive concept of this invention is explained here. The inventive concept is to transform the proximity search problem between high-dimensional vectors into a prefix matching problem for one-dimensional strings through a hierarchical spatial encoding mechanism, thereby enabling efficient and dynamic retrieval of high-dimensional vector data using mature and stable B+ tree indexing technology. Specifically, the first stage, spatial partitioning and encoding: Objective: To divide a continuous, unordered high-dimensional space into organized, hierarchical discrete units. Method: Recursively binary divide each dimension, assigning a binary bit (0 or 1) to its sub-intervals at each partition. Through multiple recursions, each finest-grained interval obtains a unique binary code, which records all path information from the root interval to that unit. Analogy: This is like dividing a country into administrative regions (provinces, cities, districts, streets) and assigning a unique postal code to each smallest administrative unit. The code itself (e.g., 110101) implicitly contains its geographical location hierarchy.

[0033] Phase Two: Alternating Combinations: Objective: To ensure that vectors that are close in position in the original high-dimensional space generate similar one-dimensional encoded strings. Method: When combining the codes of each dimension into a global string, instead of simple sequential concatenation (e.g., dimension 1 code + dimension 2 code), an alternating combination is used. That is, first take the first bit of all dimension codes, then take the second bit of all dimension codes, and so on, alternating like shuffling cards to weave them into a new binary string. Key Insight: This "shuffling" encoding ensures that if two vectors are close in any dimension, their global codes will start to be the same at a very early position (prefix). This is the core bridge for transforming high-dimensional proximity into one-dimensional string prefix similarity.

[0034] Phase 3: B+ Tree Index: Objective: To achieve efficient precise and fuzzy queries using highly mature database technology. Method: The global binary string with prefix similarity generated in the previous step is used as the primary key and input into the B+ tree to build the index. Working principle: The B+ tree sorts and stores these binary strings in lexicographical order. Due to the alternating combination encoding characteristics, vectors with the same prefix (i.e., spatially close) will have their index keys adjacent to each other in the B+ tree and will be stored in physically contiguous leaf nodes. When performing a fuzzy query (nearest neighbor search), only the encoding of the target vector needs to be calculated, its prefix is ​​matched in the B+ tree to find the starting position, and then a small range is traversed forward and backward using the linked pointers between leaf nodes. This quickly collects a large number of nearby candidate vectors, thus avoiding a brute-force scan of the entire database.

[0035] The specific implementation method is as follows: like Figure 1 The diagram shown is a flowchart of a vector database construction method provided by the present invention.

[0036] As an example, the method includes: Step S1, restricting the values ​​of each dimension of the original vector within a preset range, and normalizing the values ​​of each dimension within the preset range based on a preset scaling factor to obtain a normalized vector; Step S2, performing a recursive binary search on the preset range to obtain N sub-intervals; Step S3, assigning corresponding binary codes to the N sub-intervals based on preset encoding rules; Step S4, constructing a mapping table for each dimension to store the mapping relationship between each sub-interval and the binary code; Step S5, for the sub-intervals to which each dimension value belongs in the normalized vector, querying the binary code corresponding to the sub-interval based on the mapping table, and interleaving the binary codes of all dimensions to generate a global binary string; Step S6, using the global binary string and the original vector as a joint index key to construct a B+ tree index, based on which the B+ tree index supports insertion, deletion, modification, exact query, and fuzzy query operations on vector data.

[0037] In some feasible implementations, step S1 includes: step S101, replacing the upper limit value of the original vector that exceeds the preset range with the upper limit value, and replacing the lower limit value of the original vector that is lower than the preset range with the lower limit value; step S102, scaling the vector after value restriction processing by using a preset linear transformation formula so that all its dimension values ​​fall within a uniform normalization range.

[0038] Preferably, vector value normalization first restricts excessively large or small "abrupt" values ​​in the vector, setting them as the maximum or minimum non-abrupt values. This aims to address the problem where a sudden change in a vector value could lead to an excessively large scaling factor k, resulting in other normal values ​​and their distance differences being too small. Under the same partitioning method, the smaller the distance, the larger the Euclidean distance error. Restricting "abrupt" values ​​can solve the problem of subspace partitioning failure, ensuring the effectiveness of space partitioning. A range (-m, m) is selected for the vector values, and all element values ​​of the vector are scaled proportionally to this range by k.

[0039] In other words, since the original vector may contain "mutation values" (such as [0.1, 0.2, 5000]) that are orders of magnitude larger than those in other dimensions, if directly scaled, these mutation values ​​will "crowd out" most of the numerical space, causing other normal values ​​to be compressed into a very small interval, resulting in a loss of distinguishability. Therefore, vector value normalization can effectively avoid the problem of subspace partitioning failure. Specifically, the input is: the original vector and preset clamping upper and lower bounds (such as -m and m). The processing is: the system checks each dimension value in the vector one by one. If a value is greater than the upper bound m, it is replaced with m; if a value is less than the lower bound -m, it is replaced with -m; values ​​between [-m, m] remain unchanged. The output is: a "safe" vector that has eliminated extreme mutation values. Example: the original vector [0.5, 2.0, -1.5, 1000], with m=2.0, is clamped to get [0.5, 2.0, -1.5, 2.0]. The normalized vector obtained by scaling proportionally to a preset scaling factor of 2 is [0.25, 1.0, -0.75, 1.0].

[0040] In some feasible implementations, the termination condition of the recursive binary search partitioning is reaching a preset partitioning depth; the partitioning depth determines the number of intervals and the encoding length of each dimension; the partitioning depth is configured in a balanced manner based on the required precision and available storage resources. By adjusting the depth of the recursive binary search partitioning, the balance between system precision and storage overhead is controlled; when the partitioning depth increases, the spatial partitioning precision improves, and the corresponding mapping table size increases; when the partitioning depth decreases, the spatial partitioning precision decreases, and the corresponding mapping table size decreases. The relationship between the number of spatial partitions, precision, and the number of skip list elements is shown in Table 1.

[0041] Table 1: accuracy Number of divisions for each dimension Number of elements in the skip list 1 1 2 2 2 4 0.25 3 8 0.125 4 16 0.0625 5 32 0.03125 6 64 In some feasible implementations, combined with Figure 2As shown, step S2 involves recursively dividing the preset range into N sub-intervals using a binary search method, and the preset encoding rule is that during each binary search division, a binary bit 0 is assigned to the left interval and a binary bit 1 is assigned to the right interval. The final binary code of a sub-interval is the sequential concatenation of all assigned binary bits along the path from the initial interval to that interval. Specifically, taking m=1 and 3 divisions as an example: Recursively binary search is performed on the range [-1, 1] of dimension 1: First division: The interval [-1, 1] is divided into [-1, 0) and [0, 1], with codes of 0 and 1 respectively. Second division: The interval [-1, 0) is divided into [-1, -0.5) and [-0.5, 0), with codes of 00 and 01 respectively; the interval [0, 1] is divided into [0, 0.5) and [0.5, 1], with codes of 10 and 11 respectively. The third partitioning: Continue partitioning each sub-interval. For example, the interval [0.5, 1] ​​is divided into [0.5, 0.75) and [0.75, 1], with codes 110 and 111 respectively. This process is repeated continuously, with each partitioning making the intervals finer and increasing the code length by 1 bit, until the preset partitioning depth is reached. Output: An "interval tree", where each leaf node represents a finest-grained interval and its corresponding complete binary code.

[0042] In some feasible implementations, the mapping table is a skip list structure; the skip list dynamically records the mapping relationship between the interval range of the partitioned sub-intervals and their binary codes through a multi-level index structure; the skip list supports interval query operations with logarithmic time complexity and supports efficient insertion and deletion operations. Specifically, the skip list construction process includes: Input: All the finest-grained intervals generated by the above recursive partitioning and their binary codes. Data structure: The skip list is a data structure that can replace the balanced tree, consisting of multiple linked lists. The bottom layer is an ordered linked list containing all interval nodes, sorted by the lower bound of the interval. The upper layer is a "fast track" of the bottom linked list, containing fewer nodes. Each node has multiple pointers, pointing to the next node in the same level and the corresponding node in the lower level. Construction process: The system inserts all generated intervals (such as (-1, -0.75): 000, (-0.75, -0.5): 001, ...) into the skip list in sequence as nodes. The insertion level is randomly determined when the node appears in the higher-level index. Function: This skip list becomes a "dictionary" or "map" for this dimension. Given a numerical value, the query process starts from the highest level of the skip list, quickly skipping numerous intervals that cannot contain the value, progressively locating downwards until the precise interval containing the value is found at the lowest level, and its binary code is returned. This process has a time complexity of O(log n), making it highly efficient. Based on the examples of steps S2 and S3 above, the generated skip list can be represented as: {(-1,-0.75):000,(-0.75,-0.5):001,(-0.5,-0.25):010,(-0.25,0):011,(0,0.25):100,(0.25,0.5):101,(0.5,0.75):110,(0.75,1):111}.

[0043] In some feasible implementations, the interleaving combination method in step S5 is as follows: if the vector has M dimensions, the encoding of each dimension is as follows: , ,......, Each code has a length of L. When combining, take the first bit of all dimension codes in sequence, arrange them in dimension order, then take the second bit of all dimension codes in sequence, and so on, until all L bits are taken. Concatenate all the binary bits obtained in this order to form the global binary string.

[0044] Preferably, the goal of step S5 is to convert a high-dimensional vector into a one-dimensional binary string, so that vectors that are similar in the original high-dimensional space have a longer common prefix in their binary strings. Implementation process and mechanism: 1. Dimensional encoding query: Input: A normalized vector, and a skip list corresponding to each dimension. Processing: The system traverses each dimension of the vector. For the value of the i-th dimension, the system queries the skip list corresponding to the i-th dimension. The skip list returns the binary encoding of the interval containing the value. Output: A list of binary encodings, where each element corresponds to the encoding of one dimension of the vector. For example, for a two-dimensional vector [0.6, -0.2], the query yields dimension 1 encoding 110 and dimension 2 encoding 010. 2. Alternating combination: Input: The list of binary encodings for each dimension obtained above. Processing: The system does not simply concatenate the encodings of all dimensions in order (e.g., 110010). Instead, a "weaving" or "shuffling" strategy is used: first, the first bit of each dimension's encoding is taken and arranged in dimensional order; then, the second bit of each dimension's encoding is taken and arranged in dimensional order; and so on, until all bits are taken. Output: a single, compact binary string. Continuing the example: Dimension 1 encoding: 110; Dimension 2 encoding: 010; Alternating combination process: Take the first bit of each dimension: 1 (from dimension 1), 0 (from dimension 2) → 10; Take the second bit of each dimension: 1 (from dimension 1), 1 (from dimension 2) → 11; Take the third bit of each dimension: 0 (from dimension 1), 0 (from dimension 2) → 00; Final binary string: 101100. This method ensures that every prefix bit of the final binary string contains information from all dimensions. If two vectors are close in any dimension, their encodings will begin to resemble each other early on. This perfectly transforms the high-dimensional proximity problem into a string prefix matching problem.

[0045] In some feasible implementations, the fuzzy query operation specifically includes: calculating the global binary string of the target vector; performing a prefix matching query based on the binary string in the B+ tree index to locate the leaf node region storing similar codes; and using the chained pointers between the leaf nodes of the B+ tree, traversing the leaf node and its multiple adjacent leaf nodes to collect a set of candidate vectors.

[0046] In some feasible implementations, the fuzzy query operation further includes a reordering step: after collecting the candidate vector set, calculating the precise distance between each candidate vector in the candidate vector set and the query target vector; sorting the candidate vectors according to the precise distance, and returning the K vectors with the smallest distance as the final query result.

[0047] The preferred implementation steps for B+ tree indexing and operations are as follows: The goal is to leverage mature database indexing technology to achieve efficient storage and retrieval of vector data, especially fast proximity search. Implementation process and mechanism: 1. B+ tree index construction: Input: Key-value pairs (binary string, original vector) generated from all vectors. Processing: The system inserts the key-value pairs into a B+ tree using the binary string as the primary key (or as a combined primary key with the original vector). The B+ tree sorts and stores all these keys according to the lexicographical order of the binary string (i.e., the natural order of the strings). Key characteristic: Because binary strings preserve spatial locality (similar vectors have similar prefixes), these vectors are physically stored together in the leaf nodes of the B+ tree. 2. Operation implementation: Exact query: Given a target vector, calculate its binary string and perform an exact key-value search in the B+ tree, with extremely high efficiency (O(log n)). Fuzzy query (proximity search): Step 1: Calculate the binary string of the target vector (e.g., 101100). Step 2: Perform a prefix search in the B+ tree to find keys with the same prefix. The B+ tree can quickly locate the leaf nodes storing these keys. Step 3: Utilizing the doubly linked list property of the B+ tree's leaf nodes, traverse a certain number of leaf nodes in both forward and backward directions from this location. Step 4: During the traversal, collect the encountered vectors as a candidate set. Because these vectors are physically adjacent in the B+ tree, their encoding (i.e., spatial location) is close to the target vector. Step 5 (Reordering): To obtain the most accurate results, the system can calculate the precise Euclidean distance between each vector and the target vector on this significantly reduced candidate set, and then return the K vectors with the smallest distance. Through this series of ingenious steps, this invention successfully transforms the complex problem of high-dimensional vector proximity search into a mature prefix matching and range traversal problem on a B+ tree index, thereby achieving efficient, dynamic, and scalable vector database operations.

[0048] In some feasible implementations, for ease of understanding, specific examples are provided here: System parameters are: clamping range: [-2, 2]; normalization target range: [-1, 1]; spatial partitioning depth: 3 (each dimension generates 8 intervals); vector dimension: 2-dimensional (for ease of understanding). The original set of vectors to be processed is: Vector A: [0.5, 2.0] (embedding vector of document A); Vector B: [0.6, -0.2] (embedding vector of document B); Vector C: [0.4, 0.8] (embedding vector of document C); Vector D: [-1.8, 1.5] (embedding vector of document D). Vector normalization: Vector A: [0.5, 2.0] → [0.5, 2.0] (both within [-2,2], no change); Vector B: [0.6, -0.2] → [0.6, -0.2] (no change); Vector C: [0.4, 0.8] → [0.4, 0.8] (no change); Vector D: [-1.8, 1.5] → [-1.8, 1.5] (no change). Proportional scaling: Mapping the range [-2,2] to [-1,1] using a linear transformation formula: Scaling factor calculation: Normalized results: Vector A: [0.5×0.5, 2.0×0.5]=[0.25, 1.0]; Vector B: [0.6×0.5, -0.2×0.5]=[0.3, -0.1]; Vector C: [0.4×0.5, 0.8×0.5]=[0.2, 0.4]; Vector D: [-1.8×0.5, 1.5×0.5 + 0]=[-0.9, 0.75]. Second step: High-dimensional space partitioning and skip list construction: The range [-1, 1] undergoes 3 binary partitions: First division: [-1,0)→0,[0,1]→1; Second division: [-1,-0.5)→00, [-0.5,0)→01; [0,0.5)→10, [0.5,1]→11; Third partition: [-1,-0.75)→000, [-0.75,-0.5)→001; [-0.5,-0.25)→010, [-0.25,0)→011; [0,0.25)→100, [0.25,0.5)→101; [0.5,0.75)→110, [0.75,1]→111. Example of skip list content (dimension 1): Skip list node: (-1.0, -0.75) → "000"; Skip list node: (-0.75, -0.5) → "001"; Skip list node: (-0.5, -0.25) → "010"; Skip list node: (-0.25, 0.0) → "011"; Skip list node: (0.0, 0.25) → "100"; Skip list node: (0.25, 0.5) → "101"; Skip list node: (0.5, 0.75) → "110"; Skip list node: (0.75, 1.0) → "111".

[0049] Step 3: Vector Encoding: Vector A: [0.25, 1.0]; Dimension 1: 0.25 → Interval [0.25, 0.5) → Encoding 101; Dimension 2: 1.0 → Interval [0.75, 1.0] → Encoding 111; Vector B: [0.3, -0.1]; Dimension 1: 0.3 → Interval [0.25, 0.5) → Encoding 101; Dimension 2: -0.1 → Interval [-0.25, 0.0) → Encoding 011; Vector C: [0.2, 0.4]; Dimension 1: 0.2 → Interval [0.0, 0.25) → Encoding 100; Dimension 2: 0.4 → Interval [0.25, 0.5) → Encoding 101; Vector D: [-0.9, 0.75]; Dimension 1: -0.9 → Interval [-1.0, -0.75) → Encoding 000; Dimension 2: 0.75 → Interval [0.75, 1.0] → Encoding 111; Vector A encoding process: Dimension 1 encoding: 101; Dimension 2 encoding: 111; Alternating combination: Take all 1st bits → 1,1; Take all 2nd bits → 0,1; Take all 3rd bits → 1,1; Final binary string: 110111; Vector B encoding process: Dimension 1: 101, Dimension 2: 011, Alternating: 1st bit 1,0; 2nd bit 0,1; 3rd bit 1,1; Final binary string: 100111; Vector C encoding process: Dimension 1: 100, Dimension 2: 101; Alternating: 1st bit 1,1; 2nd bit 0,0; 3rd bit 0,1; final binary string: 110001; Vector D encoding process: Dimension 1: 000, Dimension 2: 111; Alternating: 1st bit 0,1; 2nd bit 0,1; 3rd bit 0,1; final binary string: 010101.

[0050] Step 4: B+ Tree Index Construction and Querying Key: 010101 (vector D) → Value: {Original vector: [-1.8, 1.5], Document: D}; Key: 100111 (vector B) → Value: {Original vector: [0.6, -0.2], Document: B}; Key: 110001 (vector C) → Value: {Original vector: [0.4, 0.8], Document: C}; Key: 110111(vector A) → Value: {Original vector: [0.5,2.0], Document: A}.

[0051] Fuzzy Query Example: Finding Documents Similar to Vector B: Query Target: Find the vector most similar to vector B [0.6, -0.2]: Step 1: Calculate the binary string of the query vector: The binary string of vector B is: 100111; Step 2: Perform prefix matching in the B+ tree: Locate the leaf node containing the key 100111, which contains: 100111 (vector B); Step 3: Traverse adjacent leaf nodes to collect candidate sets: Forward traversal: Find 010101 (vector D); Backward traversal: Find 110001 (vector C) and 110111 (vector A); Candidate set: {vector D, vector B, vector C, vector A}; Step 4: Calculate the exact distance and sort: Calculate the Euclidean distance to the target vector B on the candidate set: Distance to vector D: approximately 2.42; Distance to vector B: 0 (itself); Distance to vector C: approximately 1.02; Distance to vector A: approximately 1.20. The final results (sorted by similarity) are: 1. Vector B (distance 0) - Document B; 2. Vector C (distance 1.02) - Document C; 3. Vector A (distance 1.20) - Document A.

[0052] The above implementation method achieves a breakthrough in query efficiency by transforming the problem, reducing computational complexity from... It reduces computation time to O(log n); leverages the dynamic characteristics of skip lists and B+ trees to support real-time data updates; achieves a flexible balance between precision and resources by adjusting partition depth; and ensures system stability and scalability by relying on mature index structures, providing an efficient solution for large-scale high-dimensional data applications.

[0053] Example 2 Please see Figure 3 This embodiment provides a schematic diagram of a vector database construction system.

[0054] As an example, the system is implemented using the vector database construction method described in Embodiment 1, and the system includes: The normalization module 310 is suitable for limiting the values ​​of each dimension of the original vector within a preset range, and normalizing the values ​​of each dimension within the preset range based on a preset scaling factor to obtain a normalized vector.

[0055] The spatial partitioning and encoding management module 320 is suitable for performing a recursive binary division on the preset range to obtain N partitioned sub-intervals; assigning corresponding binary codes to the N partitioned sub-intervals based on preset encoding rules; and constructing a mapping table for each dimension to store the mapping relationship between each partitioned sub-interval and the binary code.

[0056] The vector encoding module 330 is suitable for obtaining the sub-intervals to which each dimension value belongs for the normalized vector, querying the binary code corresponding to the sub-interval based on the mapping table, and interleaving and combining the binary codes of all dimensions to generate a global binary string.

[0057] B+ tree storage engine module 340 is suitable for using the global binary string and the original vector as a combined index key to construct a B+ tree index, and supports insertion, deletion, modification, exact query and fuzzy query operations on vector data based on the B+ tree index.

[0058] It is not difficult to see that this embodiment is a system implementation corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0059] It is worth mentioning that all modules involved in this embodiment are logical units. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to the technical problem proposed by the invention; however, this does not mean that other units are absent from this embodiment.

[0060] Example 3 Please see Figure 4 The present invention also provides an electronic device, including: a memory and a processor; the memory stores at least one program instruction; the processor loads and executes the at least one program instruction to implement the vector database construction method provided in Embodiment 1.

[0061] The memory 702 and processor 701 are connected via a bus, which may include any number of interconnecting buses and bridges, connecting various circuits of one or more processors 701 and memory 702 together. The bus may also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 701 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 701.

[0062] Processor 701 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 702 can be used to store data used by processor 701 during operation.

[0063] Example 4 This invention also proposes a storage medium storing a vector database construction method. When the vector database construction program is executed by a processor, it implements the steps of the vector database construction method described above. Since this storage medium employs all the technical solutions of the above embodiments, it possesses at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be elaborated upon here.

[0064] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, based on the guidance provided in this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for constructing a vector database, characterized in that, The method includes: Step S1: Limit the values ​​of each dimension of the original vector to a preset range, and normalize the values ​​of each dimension within the preset range based on a preset scaling factor to obtain a normalized vector. Step S2: Perform a recursive binary search on the preset range to obtain N sub-intervals; Step S3: Assign corresponding binary codes to the N sub-intervals based on preset encoding rules; Step S4: Construct a mapping table for each dimension to store the mapping relationship between each sub-interval and the binary code; Step S5: For the normalized vector, obtain the partition sub-intervals to which each dimension value belongs, query the binary code corresponding to the partition sub-interval based on the mapping table, and interleave and combine the binary codes of all dimensions to generate a global binary string; Step S6: Use the global binary string and the original vector as a combined index key to construct a B+ tree index. Based on the B+ tree index, the insertion, deletion, modification, exact query and fuzzy query operations of the vector data are supported.

2. The vector database construction method according to claim 1, characterized in that, Step S1 includes: Step S101: Replace the dimension values ​​in the original vector that exceed the upper limit value of the preset range with the upper limit value, and replace the dimension values ​​that are lower than the lower limit value of the preset range with the lower limit value; Step S102: Scale the vector after value restriction processing using a preset linear transformation formula so that all its dimension values ​​fall within a uniform normalization range.

3. The vector database construction method according to claim 1, characterized in that, The recursive binary division terminates when a preset division depth is reached; the division depth determines the number of intervals and the encoding length for each dimension; the division depth is configured in a balanced manner based on the required precision and available storage resources.

4. The vector database construction method according to claim 3, characterized in that, The method also includes a precision adjustment step: By adjusting the depth of the recursive binary search partition, the balance between system accuracy and storage overhead is achieved. As the partitioning depth increases, the spatial partitioning accuracy improves, and the corresponding mapping table size increases. When the partitioning depth decreases, the spatial partitioning accuracy decreases, and the corresponding mapping table size decreases.

5. The vector database construction method according to claim 1, characterized in that, The preset encoding rule is that during each binary division, a binary bit 0 is assigned to the left interval and a binary bit 1 is assigned to the right interval; the final binary code of a sub-interval is the sequential concatenation of all the assigned binary bits along the path from the initial interval to the current interval.

6. The vector database construction method according to claim 1, characterized in that, The mapping table is a skip list structure; the skip list dynamically records the mapping relationship between the range of the divided sub-intervals and the binary code through a multi-level index structure; the skip list supports interval query operations with logarithmic time complexity, and supports efficient insertion and deletion operations.

7. The vector database construction method according to claim 1, characterized in that, The interleaving combination method described in step S5 is as follows: If the vector has M dimensions, the codes for each dimension are respectively... , ,......, Each code has a length of L. When combining, take the first bit of all dimension codes in sequence, arrange them in dimension order, then take the second bit of all dimension codes in sequence, and so on, until all L bits are taken. Concatenate all the binary bits obtained in this order to form the global binary string.

8. The vector database construction method according to claim 1, characterized in that, The fuzzy query operation specifically includes: calculating the global binary string of the target vector; performing a prefix matching query based on the binary string in the B+ tree index to locate the leaf node region storing similar codes; and using the chained pointers between the leaf nodes of the B+ tree, traversing the leaf node and its multiple adjacent leaf nodes to collect a set of candidate vectors.

9. The vector database construction method according to claim 1, characterized in that, The fuzzy query operation also includes a reordering step: after collecting the candidate vector set, calculate the precise distance between each candidate vector in the candidate vector set and the query target vector; sort the candidate vectors according to the precise distance, and return the K vectors with the smallest distance as the final query result.

10. A vector database construction system, wherein the system is implemented using the vector database construction method according to any one of claims 1-9, characterized in that, The system includes: The normalization module is suitable for limiting the values ​​of each dimension of the original vector within a preset range, and normalizing the values ​​of each dimension within the preset range based on a preset scaling factor to obtain a normalized vector. The spatial partitioning and encoding management module is suitable for performing a recursive binary search on the preset range to obtain N partitioned sub-intervals; assigning corresponding binary codes to the N partitioned sub-intervals based on preset encoding rules; and constructing a mapping table for each dimension to store the mapping relationship between each partitioned sub-interval and the binary code. The vector encoding module is suitable for obtaining the sub-intervals to which each dimension value belongs for the normalized vector, querying the binary code corresponding to the sub-interval based on the mapping table, and interleaving and combining the binary codes of all dimensions to generate a global binary string. The B+ tree storage engine module is suitable for constructing a B+ tree index by using the global binary string and the original vector as a combined index key. Based on the B+ tree index, it supports insertion, deletion, modification, exact query, and fuzzy query operations on vector data.

Images