Time sequence vector database-oriented retrieval method
By constructing a timestamp graph and a greedy wandering algorithm to update the neighbor list, the problem of nearest neighbor retrieval of the time series vector database in high-dimensional vector data is solved, efficient and accurate time-constrained retrieval is achieved, and memory usage is reduced.
Patent Information
- Application Number
- CN202510613358.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-13
- Publication Date
- 2025-08-08
AI Technical Summary
Existing methods are difficult to achieve efficient and accurate nearest retrieval of timing vector databases in high-dimensional vector data, especially in scenarios where vector update speed is fast and query time constraints are required.
The time stamp graph structure index is designed, and the neighbor list is updated through the greedy wandering algorithm, combining the timestamp graph and historical neighbor list to achieve efficient and accurate vector nearest neighbor retrieval.
Improves the accuracy of retrieval, reduces the memory usage of indexes, and can adapt to scenes of fast vector updates.
Smart Images
Figure CN120448404A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of vector databases, relates to a retrieval problem for high-dimensional vector data with time constraints, and specifically to a retrieval method for time series vector databases. Background Art
[0002] Vector databases are used to store and manage the high-dimensional vectors generated by embedding various unstructured data (such as text, images, and audio). Vector databases play a crucial role in emerging AI applications, such as managing external knowledge bases and providing large models with relevant information to mitigate large model hallucinations and enhance information accuracy. Retrieval of relevant information is typically achieved through high-dimensional vector nearest neighbor retrieval: knowledge base text and query statements are encoded into high-dimensional vectors, and semantically similar documents are searched through nearest neighbor retrieval between these vectors.
[0003] In addition to semantic relevance, documents in a knowledge base are also subject to time constraints: each document has a certain validity period. For example, knowledge documents describing company market capitalizations and weather information are constantly updated over time. To improve the relevance of retrieved documents, both semantic relevance and time relevance need to be considered.
[0004] Existing methods have difficulty implementing neighbor retrieval in time series vector databases. One type of existing work constructs a tree index for all valid objects at each timestamp, and searches the tree index at the query timestamp t to find the neighbors that meet the time constraint. However, this type of method is only applicable to low-dimensional data (such as spatiotemporal data) and is difficult to migrate to high-dimensional vectors due to the limitations of the "curse of dimensionality" phenomenon. Another type of existing work designs graph structure indexes for massive high-dimensional vector data to support efficient retrieval. However, this type of method is mainly designed for static vector datasets and has difficulty maintaining the storage structure and supporting the retrieval of valid vectors at a given timestamp in highly dynamic scenarios.
[0005] In summary, nearest neighbor retrieval for time series vector data is widely used in artificial intelligence applications such as large models. However, existing methods struggle to address the challenges of high vector dimensionality, rapid vector updates, and time-constrained queries. Therefore, it is necessary to design indexing and retrieval methods specifically for this problem to achieve efficient and accurate nearest neighbor retrieval in time series vector databases. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this paper designs a timestamp graph, an index structure that simultaneously considers vector neighbor relationships and time validity, for retrieval in time-series vector databases. This paper focuses on the time-constrained vector nearest neighbor retrieval problem: given a high-dimensional query vector and a timestamp t, searching a time-series vector database for the closest vector to the high-dimensional query vector that is valid at the given timestamp t.
[0007] The retrieval method for time series vector database is divided into the following steps:
[0008] Step 1: Build a timestamp graph;
[0009] For a time series vector database composed of high-dimensional vectors, a graph structure index is constructed based on time to obtain a timestamp graph. The specific construction process is as follows:
[0010] First, a graph structure index is constructed for all vectors valid at each moment, and then the graph structure indexes of each moment are merged into a single graph index to form a timestamp graph.
[0011] Set the initialization parameters of the timestamp graph, that is, the number of neighbors parameter M.
[0012] Step 2: When a new vector point is added to the timestamp graph at time t, the historical neighbor list of the new vector point at that time is found by greedy walking on the timestamp graph;
[0013] When a new vector point is added to the timestamp graph at time t, M neighbors are found on the timestamp graph through greedy walks, and these neighbors are saved as the historical neighbor list of the new vector point at time t. Specifically:
[0014] A starting point ep is randomly selected in the initial timestamp graph TG. When a new vector point v is added to the timestamp graph at time t, the vector starts to wander from the starting point ep. When it reaches point u, it traverses all of u's neighbors. If there is a neighbor u' of u such that the distance between u' and v is closer than the distance between u and v, it wanders to point u'. If there are multiple neighbors u' closer to v, it preferentially wanders to the neighbor closest to v.
[0015] If the distances between u's neighbors and v are all greater than the distance between u and v, the point has been reached. Further wandering will no longer reduce the distance between the currently visited point and v. The visited point is called v′. Next, the M points closest to v are selected from all two-hop neighbors of v′. These M vectors are used as the neighbors of the new vector point v. v is then connected to these M neighbors to form the historical neighbor list of the new vector point v.
[0016] Afterwards, the historical neighbor list of each neighbor point in the historical neighbor list of the new vector point at time t is updated.
[0017] For each neighbor point w of v, perform the following operations: add v to the neighbor list of each neighbor point w. Then, iterate through the new neighbor list of each neighbor point w, retaining the M points closest to w as neighbors, and complete the update of the historical neighbor list of each neighbor point of vector point v at time t. Then, insert the updated neighbor list at time t into the historical neighbor list of each neighbor point w.
[0018] Step 3: When an existing vector point p expires at time t, p is disconnected from its neighbors on the timestamp graph TG, and the historical neighbor lists of each neighbor point of p are updated;
[0019] The details are as follows:
[0020] First, remove p from the neighbor list of each neighbor point w. Then, calculate the number of w's neighbors and determine whether the number of w's neighbors is greater than M / 2. If so, directly use these neighbors as the neighbor list of w at time t. Otherwise, if the number of w's neighbors is less than or equal to M / 2, run the greedy walk algorithm on the TG to find a neighbor list of w that contains M neighbors at time t. Finally, add the updated neighbor list at time t to w's historical neighbor list.
[0021] Step 4: When querying the nearest neighbor of vector point q at time t, perform a greedy walk with time constraints on the timestamp graph. According to the historical neighbor list at that time, find the nearest neighbor of vector point q that is valid at time t, that is, walk towards the query q in the vector that meets time t.
[0022] The details are as follows:
[0023] Starting from a random starting point ep, the algorithm repeats the following steps. When it reaches point u, it uses u's historical neighbor list to find all valid neighbors at time t. Iterate through these neighboring points. If a neighbor u' brings u' and q closer than u and q, it reaches point u'. Repeat this process until the distances between u's neighbors and v are all greater than the distance between u and v. The point visited at this point is called v'. From all two-hop neighbors of v', it selects the M points closest to q. These M vectors are considered the nearest neighbors of vector q. In the above algorithm, all points visited at a given time t are considered valid.
[0024] The advantages of the present invention are:
[0025] 1. The present invention uses a greedy walk with time constraints to find the nearest neighbor vectors that are valid at a specified timestamp, ensuring the accuracy of the nearest neighbor retrieval in the time series vector database.
[0026] 2. The present invention has a fast update speed for the timestamp map. When accepting an update, only M local points of the timestamp map need to be changed. This can adapt to the scenario where a large amount of vector data is updated in a short period of time.
[0027] 3. The timestamp graph in the present invention merges the graph structure indexes on each timestamp through the historical neighbor list mechanism, reducing the memory usage of the index. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 Schematic diagram of the flow of the time series vector retrieval method based on the timestamp graph of the present invention;
[0029] Figure 2 Schematic diagram of a greedy walk with time constraints according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] The present invention provides a time series vector database retrieval method based on a timestamp graph, which is applicable to dynamic time series vector database scenarios and realizes efficient and accurate time-constrained vector retrieval.
[0031] like Figure 1 The specific steps are as follows:
[0032] Step 1: Initialize the timestamp graph TG and select an arbitrary point from the vector database as the starting point ep. Set the initialization parameters of the timestamp graph, that is, the number of neighbors parameter M.
[0033] Step 2: When a new vector point v is added to the timestamp graph at time t, a greedy walk neighbor search operation is performed on the timestamp graph TG to find M neighbors, connect v to these M neighbors, and update the historical neighbor lists of these neighbors.
[0034] Step 3: When an existing vector point v expires at time t, v is disconnected from its neighbors on the timestamp graph TG, and the historical neighbor list of v's neighbors is updated.
[0035] exist Figure 2 In the example shown, it is assumed that the number of neighbors M=4.
[0036] At time t1, point o is added to the database, and the four points {u1, u2, u3, u4} are found as neighbors through greedy walk, and the neighbors at that moment are recorded in the historical neighbor list shown in Table 1.
[0037] At time t2, o's neighbor point u4 expires. At this time, o removes u4 from the neighbor list and adds the neighbor list {u1, u2, u3} at time t2 to the historical neighbor list.
[0038] At time t3, u5 is added to the database and u5 selects o as one of its M neighbors. At this time, o adds u5 to the neighbor list and adds the neighbors {u1, u2, u3, u5} at this moment to the historical neighbor list.
[0039] At t4 and t5, points u3 and u5 expire successively. At this time, there are only two points left in o's neighbor list. At this time, it is necessary to rerun the search algorithm to find M points as neighbors. The neighbor list at t5 is {u1, u2, u6, u8}.
[0040] At time t6, u9 joins the database and selects o as a neighbor. At this time, o replaces u8 with u9 as one of its neighbors and adds the updated neighbor list {u1, u2, u6, u9} to the historical neighbor list.
[0041] Finally, point o expires. At this time, it is necessary to retain the historical neighbor list of point o for subsequent query.
[0042] Step 4: When querying the nearest neighbor of vector point q at time t, perform a greedy walk with time constraints on the timestamp graph to find the nearest neighbor of vector point q that is valid at time t.
[0043] exist Figure 1 In the example shown, the time-constrained greedy walk operates as follows: If a point o is reached at any time between t1 and t2 (excluding t2), its neighbors are considered to be {u1, u2, u3, u4}, and the next hop is selected from these neighbors, which is closer to the query q. If a point o is reached at any time between t2 and t3 (excluding t3), its neighbors are considered to be {u1, u2, u3}. Since u4 has expired at t2, it no longer appears in the neighbor list after t2. Repeating the above process completes the time-constrained greedy walk and finds the nearest neighbors of the vector point q.
[0044] For example, in a large-scale question-answering system based on a knowledge base, the large model needs to retrieve relevant documents from the knowledge base through vector nearest neighbor queries to answer questions. However, the documents in the knowledge base are constantly changing. To retrieve the documents most relevant to the question, both semantic relevance (achieved through vector distance calculations) and temporal relevance (achieved through query timestamp constraints) must be considered. By building a timestamp graph to manage the vectors in the knowledge base and their valid time intervals, efficient retrieval of the knowledge base is achieved, ensuring that the retrieved documents are within the specified time range.
[0045] Table 1 List of historical neighbors of o during timestamp graph update
[0046] time Historical Neighbor List <![CDATA[t1]]> <![CDATA[{u1,u2,u3,u4}]]> <![CDATA[t2]]> <![CDATA[{u1,u2,u3}]]> <![CDATA[t3]]> <![CDATA[{u1,u2,u3,u5}]]> <![CDATA[t4]]> <![CDATA[{u1,u2,u5}]]> <![CDATA[t5]]> <![CDATA[{u1,u2,u6,u8}]]> <![CDATA[t6]]> <![CDATA[{u1,u2,u6,u9}]]>
Claims
1. A retrieval method for a time series vector database, characterized in that: The following steps are involved: Step 1: For the time series vector database composed of high-dimensional vectors, a graph structure index is constructed based on time to obtain a timestamp graph; Step 2: When a new vector point is added to the timestamp graph at time t, the historical neighbor list of the new vector point at that time is found by greedy walking on the timestamp graph; When a new vector point is added to the timestamp graph at time t, M neighbors are found on the timestamp graph through greedy walks, and these neighbors are saved as the historical neighbor list of the new vector point at time t. Specifically: A starting point ep is randomly selected in the initial timestamp graph TG. When a new vector point v is added to the timestamp graph at time t, it starts to wander from the starting point ep. When it wanders to point u, it traverses all neighbors of u. If there is a neighbor u′ of u such that the distance between u′ and v is closer than the distance between u and v, it wanders to point u′; if there are multiple neighbors u′ closer to v, it preferentially wanders to the neighbor closest to v. If the distance between u's neighbor and v is greater than the distance between u and v, the point has been reached. Continuing to walk will no longer reduce the distance between the currently visited point and v. The visited point is then called v'. Then, M points closest to v are selected from all two-hop neighbors of v'. These M vectors are used as neighbors of the new vector point v. v is then connected to these M neighbors to form a historical neighbor list for the new vector point v. Afterwards, v is added to the neighbor list of each of its neighbor points w, and the neighbor list of each of v's neighbor points w at time t is updated to obtain a new historical neighbor list; Step 3: When an existing vector point p expires at time t, p is disconnected from its neighbors on the timestamp graph TG, and the historical neighbor lists of each neighbor point of p are updated; Step 4: When querying the nearest neighbor of vector point q at time t, perform a greedy walk with time constraints on the timestamp graph. According to the historical neighbor list at this time, find the nearest neighbor of vector point q that is valid at time t, that is, walk in the direction of query q in the vector that meets time t.
2. A retrieval method for a time series vector database according to claim 1, characterized in that: The build timestamp screenshot is as follows: First, a graph structure index is constructed for all vectors valid at each moment, and then the graph structure indexes of each moment are merged into a single graph index to form a timestamp graph.
3. The retrieval method for a time series vector database according to claim 2, characterized in that: When constructing the timestamp graph, set the initialization parameters of the timestamp graph, that is, the neighbor number parameter M.
4. The retrieval method for a time series vector database according to claim 1, characterized in that: At time t, the neighbor list of each neighbor point w of v is updated. The specific operations are as follows: Add v to the neighbor list of each of its neighbor points w, then traverse the new neighbor list of each neighbor point w, retain the M points closest to w as neighbors, and complete the update of the historical neighbor list of each neighbor point of vector point v at time t; then, insert the updated neighbor list at time t into the historical neighbor list of each neighbor point w.
5. The retrieval method for a time series vector database according to claim 1, characterized in that: When the vector point p expires at time t, the historical neighbor list of its neighbor points is updated as follows: First, remove p from the neighbor list of each neighbor point w, then calculate the number of neighbors of w and determine whether the number of neighbors of w is greater than M / 2. If so, directly use these neighbors as the neighbor list of w at time t; otherwise, the number of neighbors of w is less than or equal to M / 2, then run the greedy walk algorithm on TG to find the neighbor list of w that contains M neighbors at time t; finally, add the updated neighbor list at time t to the historical neighbor list of w.
6. The retrieval method for a time series vector database according to claim 1, characterized in that: Query the nearest neighbor of vector point q at time t, as follows: Starting from a random starting point ep, repeat the following operations. When you reach point u, find all valid neighbors at time t through u's historical neighbor list; traverse these neighbor points. If the neighbor u' makes the distance between u' and q closer than the distance between u and q, then walk to point u'; repeat this operation until the distance between u's neighbor points and v is greater than the distance between u and v. The point visited at this time is recorded as v'. Select M points closest to q from all two-hop neighbors of v', and use these M vectors as the nearest neighbors of vector q.