A database-based fusion index construction and query method, device and medium

Abstract

This application discloses a database-based fusion index construction and query method, device, and medium, relating to the field of data processing technology. The method includes: receiving an index creation command and parsing out vector column names and scalar column names; reading vector data corresponding to the vector column names, original scalar values ​​corresponding to the scalar column names, and row identifiers from a preset database table, and combining the vector data, original scalar values, and row identifiers into index tuples; constructing a hierarchical navigable small-world graph structure by using the index tuples as nodes based on the vector data of each index tuple; receiving a fusion query command and traversing nodes by using the entry node of the hierarchical navigable small-world graph structure as the current node to be accessed; reading the inline scalar blocks of the currently accessed node and storing the row identifiers of nodes that meet preset scalar conditions into a result candidate heap; stopping traversal when the result candidate heap contains a preset number of row identifiers, thus completing the fusion query command.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a database-based method, device, and medium for constructing and querying fusion indexes. Background Technology

[0002] In AI-driven database applications, a common query pattern is to perform vector similarity search within the result set that meets scalar filtering conditions. Currently, the PostgreSQL ecosystem primarily supports vector retrieval through the pgvector extension. For mixed queries, existing technologies fall into three categories: First, there are two-stage serial filtering schemes (such as pgvector), which first use a B-tree index to filter all row identifiers that meet the scalar conditions, then load the corresponding vector data and calculate the distance. The drawback is that the I / O and memory overhead is proportional to the size of the scalar filtering result set. When the filtering conditions are lenient, the candidate set can reach hundreds of thousands, leading to a serious performance bottleneck. Second, there are post-filtering schemes (such as ScyllaDB and Milvus), which first perform pure vector retrieval to obtain candidate nearest neighbors, and then apply scalar condition filtering. However, if the scalar conditions are highly selective, a large number of nodes recalled by the vector index but not meeting the conditions have already had their distances calculated, resulting in wasted computation. Finally, there are index association schemes (such as OceanBase and Lindorm), which associate the scalar condition results with the vector index nodes through an external Bitmap or mapping table. During the query, a set of IDs for scalar filtering needs to be pre-built, and this set needs to be queried during the vector traversal. The drawback is that additional I / O is still required to read the external mapping structure, and the scalar conditions must be pre-computed as an ID set, which cannot directly support range queries, expressions, and other general SQL predicates.

[0003] Therefore, how to store scalar attributes inline in the nodes of a vector index and use the inline scalar information to perform real-time pruning on each accessed node during a single index traversal, thereby changing the I / O and memory overhead from being related to the size of the filtered result set to being related to the final result set, and achieving efficient and general-purpose hybrid queries, has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a database-based fusion index construction and query method, device, and medium to solve the following technical problem: how to store scalar attributes inline in the nodes of a vector index, and how to use the inline scalar information to perform real-time pruning on each access node during a single index traversal, thereby changing I / O and memory overhead from being related to the size of the filtered result set to being related to the final result set, and achieving efficient and universal fusion queries.

[0005] In a first aspect, embodiments of this application provide a database-based fusion index construction and query method, the method comprising: receiving an index creation command and parsing vector column names and scalar column names from the index creation command; reading vector data corresponding to the vector column names, original scalar values ​​corresponding to the scalar column names, and row identifiers from a preset database table, and combining the vector data, original scalar values, and row identifiers into index tuples; constructing a hierarchical navigable small-world graph structure by using the index tuples as nodes based on the vector data of each index tuple; receiving a fusion query command and traversing the nodes by using the entry node of the hierarchical navigable small-world graph structure as the current node to be accessed; reading the inline scalar blocks of the current node to be accessed and storing the row identifiers of nodes that meet preset scalar conditions into a result candidate heap, stopping the traversal when the result candidate heap contains a preset number of row identifiers, thereby completing the fusion query command.

[0006] In one implementation of this application, receiving an index creation command and parsing vector column names and scalar column names from the index creation command specifically includes: splitting the syntax structure of the index creation command to separate the index definition body and the additional inclusion clause; extracting vector column identifier information used to construct the vector retrieval dimension from the index definition body and extracting scalar column identifier information of the index node from the additional inclusion clause; filtering duplicate column identifier information and organizing the vector column name sequence and scalar column name sequence according to the syntax definition order.

[0007] In one implementation of this application, the vector data corresponding to the vector column name, the original scalar value corresponding to the scalar column name, and the row identifier are read from a preset database table, and the vector data, the original scalar value, and the row identifier are combined into an index tuple. Specifically, this includes: traversing each data row in the preset database table and obtaining the row identifier of the current data row; based on the vector column name and the scalar column name, reading the complete vector data and the original field values ​​of each scalar column from the current data row, and encapsulating the original field values ​​of each scalar column into an inline scalar block; performing structured encapsulation matching on the row identifier, the complete vector data, and the inline scalar block, and completing field concatenation according to a unified storage format to obtain the index tuple.

[0008] In one implementation of this application, a hierarchical navigable small-world graph structure is constructed by using the vector data of each index tuple as nodes. Specifically, this includes: generating a hierarchy for the current index tuple to be inserted, and using this hierarchy as the highest level in the hierarchical navigable small-world graph structure where the current index tuple appears; starting from the current highest level of the hierarchical navigable small-world graph structure, searching downwards layer by layer, and using a vector distance metric to find the node closest to the vector data of the current index tuple from the existing nodes of the current highest level; when the search reaches the hierarchy, inserting the current index tuple as a new node into the hierarchy and all levels below it, and establishing bidirectional neighbor connections between the new node and the nearest node in each level; and pruning the neighbor list of the new node in each level based on a preset upper limit for the number of neighbor connections, retaining the nearest neighbor node to complete the construction of the hierarchical navigable small-world graph structure.

[0009] In one implementation of this application, based on a preset upper limit for the number of neighbor connections, the neighbor list of a new node in each level is pruned to retain the nearest neighbor node. Specifically, this includes: traversing all neighbor associations established by the new node in the corresponding level and calculating the vector space distance between the new node and each neighbor association; sorting all neighbor associations in ascending order of vector space distance, and selecting the top-ranked neighbor nodes according to the preset neighbor connection number constraint and retaining them in the neighbor list; removing distant neighbor associations that exceed the number constraint, and synchronously updating the reverse association topology on the side of the removed neighbor to complete the filtering and pruning of the neighbor list of new nodes in all levels.

[0010] In one implementation of this application, receiving a mixed query command and traversing the node with the entry node of the hierarchical navigable small world graph structure as the current node to be visited specifically includes: parsing the mixed query command to extract the query vector, scalar filtering predicate, and limiting the number of returned row identifiers; initializing an empty queue to be explored and an empty result candidate heap, and adding the global entry node of the hierarchical navigable small world graph structure to the queue to be explored; adding a node to the queue to be explored as the current node to be visited, and traversing the hierarchical navigable small world graph structure.

[0011] In one implementation of this application, the inline scalar block of the currently accessed node is read, and the row identifiers of nodes that meet the preset scalar conditions are stored in the result candidate heap. Traversal stops when the result candidate heap contains a preset number of row identifiers to complete the mixed query command. Specifically, this includes: reading the original scalar value stored in the inline scalar block of the currently accessed node, substituting the original scalar value into the scalar filtering predicate for evaluation to obtain a Boolean judgment result; if the Boolean judgment result is false, the currently accessed node is discarded; if the Boolean judgment result is true, the inline stored vector data of the currently accessed node is read, and the distance between the inline stored vector data and the query vector in the mixed query command is calculated to obtain a distance value; based on the distance value, the row identifier of the currently accessed node is inserted into the result candidate heap, and all neighbor nodes of the currently accessed node are added to the exploration queue until the exploration queue is empty and / or the result candidate heap contains a preset number of row identifiers, at which point traversal stops.

[0012] In one implementation of this application, the row identifier of the currently visited node is inserted into the result candidate heap based on the distance value. Specifically, this includes: if the result candidate heap is not full, the row identifier of the currently visited node is directly inserted into the result candidate heap; if the result candidate heap is full and the distance value is less than the maximum distance value in the heap, the row identifier of the currently visited node is used to replace the row identifier with the largest distance in the heap.

[0013] Secondly, embodiments of this application also provide a database-based fusion index construction and query device, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: receive an index creation command and parse vector column names and scalar column names from the index creation command; read vector data corresponding to the vector column names, original scalar values ​​corresponding to the scalar column names, and row identifiers from a preset database table, and combine the vector data, original scalar values, and row identifiers into index tuples; based on the vector data of each index tuple, use the index tuples as nodes to construct a hierarchical navigable small-world graph structure; receive a fusion query command and traverse nodes using the entry node of the hierarchical navigable small-world graph structure as the current node to be accessed; read the inline scalar blocks of the current node to be accessed and store the row identifiers of nodes that meet preset scalar conditions into a result candidate heap, stopping traversal when the result candidate heap contains a preset number of row identifiers, thereby completing the fusion query command.

[0014] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for constructing and querying a fusion index based on a database, storing computer-executable instructions. These computer-executable instructions are configured to: receive an index creation command and parse vector column names and scalar column names from the index creation command; read vector data corresponding to the vector column names, original scalar values ​​corresponding to the scalar column names, and row identifiers from a preset database table, and combine the vector data, original scalar values, and row identifiers into index tuples; based on the vector data of each index tuple, use the index tuples as nodes to construct a hierarchical navigable small-world graph structure; receive a fusion query command and traverse nodes using the entry node of the hierarchical navigable small-world graph structure as the current node to be accessed; read the inline scalar block of the currently accessed node and store the row identifiers of nodes that meet preset scalar conditions into a result candidate heap; stop traversing when the result candidate heap contains a preset number of row identifiers to complete the fusion query command.

[0015] This application provides a database-based fusion index construction and query method, device, and medium, which has the following beneficial effects: Row identifiers, vector data, and original scalar values ​​are uniformly encapsulated into fusion index tuples and embedded within HNSW graph nodes, physically integrating scalar attributes and vector data into the same index structure. This eliminates the need for pre-screening with additional scalar indexes, thus removing the data isolation and call fragmentation issues between the two types of indexes from the underlying architecture. Furthermore, the fusion index tuples are inlined with scalar blocks, synchronously completing scalar condition verification during the single-stage traversal of the HNSW graph. This eliminates the need for pre-screening of the entire scalar candidate set and batch loading of a large number of irrelevant vectors. I / O overhead is only related to the size of the final returned result, fundamentally reducing invalid disk reads and solving the I / O blocking bottleneck in large candidate set scenarios. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a database-based fusion index construction and query method provided in this application embodiment; Figure 2 This is a schematic diagram of the internal structure of a database-based fusion index construction and query device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] This application provides a database-based fusion index construction and query method, device, and medium to solve the following technical problem: how to store scalar attributes inline in the nodes of a vector index, and how to use the inline scalar information to perform real-time pruning on each access node during a single index traversal, thereby changing I / O and memory overhead from being related to the size of the filtered result set to being related to the final result set, and achieving efficient and universal fusion queries.

[0019] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0020] Figure 1 This document provides a flowchart of a database-based fusion index construction and query method as an embodiment of this application. Figure 1 As shown in the figure, the database-based fusion index construction and query method provided in this application embodiment specifically includes the following steps: Step 10: Receive the create index command and parse the vector column names and scalar column names from the create index command.

[0021] As an optional embodiment, receiving an index creation command and parsing vector column names and scalar column names from the index creation command may specifically include: Step 101: splitting the syntax structure of the index creation command to separate the index definition body and the additional inclusion clause.

[0022] In this step, the database's built-in lexical parsing mechanism is used to segment the entire index creation statement word by word and construct a syntax tree. Following standard SQL syntax rules, the statement's nested syntax and semantic structure are broken down layer by layer. During syntax parsing, the syntax range responsible for index type declaration, vector column definition, and operator specification is precisely defined by identifying index creation-specific syntax identifiers, index access method keywords, and vector field constraint syntax segments. This delineates the syntax boundaries of the main index definition. Simultaneously, the keyword features of the INCLUDE clause in the database syntax specification are matched, and subsequent clause syntax fragments belonging to the index creation statement are traced back to identify clause start identifiers and statement end boundaries, completely extracting the additional inclusion clause portion used to declare the scalar column to be inlined. Through syntax keyword matching, syntax range boundary positioning, and semantic classification of syntax fragments, the complete index creation command is broken down into a logically independent and functionally distinct main index definition portion and an additional inclusion clause portion, achieving precise separation of the two types of syntax fragments. This provides a standardized syntax parsing foundation for the subsequent extraction of vector column information and scalar column information.

[0023] Step 102: Extract the vector column identifier information used to construct the vector retrieval dimension from the main body of the index definition, and extract the scalar column identifier information of the index node from the additional containing clause.

[0024] In this step, based on the PostgreSQL index creation syntax specification, the syntax positions used to declare vector retrieval fields in the main syntax fragment are identified. Vector data type identifiers and index operator constraint fields are matched, and vector column identifier information, which serves as the basis for constructing the hierarchical navigable small world graph structure, is accurately extracted. This identifier information is used to uniquely associate the target fields that carry high-dimensional vector data in the data table. At the same time, the extracted additional containing clauses are parsed semantically. According to the syntax rules of the fields listed inside the clauses, all scalar column identifier information included in the inline storage range of the index node is identified and extracted in sequence, preserving the inherent arrangement logic and field belonging relationship of each scalar column in the clause. The entire extraction process follows the database kernel syntax parsing rules, accurately extracting the field identifiers required for vector dimension construction and the field identifiers required for the embedded scalar attributes of the index node from two types of independent syntax fragments, providing accurate field addressing basis for subsequently reading the corresponding vector data and original scalar values ​​in the table and assembling the fused index tuples.

[0025] Step 103: Filter out duplicate column identifier information and organize the vector column name sequence and scalar column name sequence according to the syntax definition order.

[0026] In this step, after the initial extraction of vector and scalar column identifier information, redundancy checks are performed on both sets of extracted vector and scalar column identifiers. The semantic definition and field attribution of each column identifier are compared one by one to identify duplicate column identifier entries caused by redundant syntax or repeated declarations. Duplicate column identifiers with identical semantic overlap or field references are removed to prevent the same field from being repeatedly placed into the index structure during subsequent index construction. After filtering for duplicate identifiers, the order of field placement in the original syntax of the index creation command is followed. The deduplicated vector column identifiers are ordered to form an ordered sequence of vector column names. At the same time, the deduplicated scalar column identifiers are sorted and arranged according to the syntactic order in which each scalar column is listed in the appended containment clause, maintaining a layout that is completely consistent with the user-defined logic. Through duplicate identifier filtering and syntactic ordering, a vector column name sequence and a scalar column name sequence with a standardized structure, no redundancy, and a word order that fits the original creation instruction are obtained. This provides a unified and ordered field layout benchmark for subsequent reading of data table fields in a fixed order, encapsulating inline scalar blocks in index nodes, and constructing standardized fused index tuples.

[0027] Step 20: Read the vector data corresponding to the vector column name, the original scalar value corresponding to the scalar column name, and the row identifier from the preset database table, and combine the vector data, the original scalar value, and the row identifier into an index tuple.

[0028] As an optional embodiment, the vector data corresponding to the vector column name, the original scalar value corresponding to the scalar column name, and the row identifier are read from a preset database table, and the vector data, the original scalar value, and the row identifier are combined into an index tuple. Specifically, this may include: Step 201: Traversing each data row in the preset database table and obtaining the row identifier of the current data row.

[0029] In this step, firstly, according to the business semantics, the storage attributes and data type constraints of various fields are defined. Vector type fields for storing high-dimensional vector data and scalar type fields for storing structured attribute information are created respectively. Then, relying on the database's native mechanism, a physical location identifier with unique identification characteristics is assigned to each row of data, thereby constructing the database table. After construction, a full data traversal operation is performed based on the preset target database table. According to the addressing rules of the database physical storage, all data records stored in the table are accessed row by row, and the storage location of each data row is located and read in sequence. During the row-by-row traversal, for the currently accessed data row, the unique row identifier of the data row is extracted through the metadata acquisition interface of the database kernel. This row identifier is used to establish a precise mapping relationship between the index node and the original data table record, ensuring that subsequent queries and retrievals can quickly locate the original data by returning to the table through the index results.

[0030] Step 202: Based on the vector column name and scalar column name, read the complete vector data and the original field values ​​of each scalar column from the current data row, and encapsulate the original field values ​​of each scalar column into an inline scalar block.

[0031] In this step, based on the vector column name sequence and scalar column name sequence obtained from the previous parsing and regularization, the complete original vector data corresponding to the vector column is accurately addressed and read from the physical storage structure of the current data row through the data field reading interface of the PostgreSQL kernel. At the same time, the original field values ​​corresponding to each scalar column are read sequentially according to the established syntax order, fully preserving the original data type and field semantic information of the scalar fields. After reading the original field values ​​of all scalar columns one by one, the column arrangement order defined by the included clause when creating the index is strictly followed. The scattered original field values ​​of each scalar column are structured, regularized, and uniformly encapsulated. They are integrated into logically continuous and formatted inline scalar blocks according to the storage organization rules agreed upon within the index. This allows the scalar attribute data to be embedded into the fused index tuple in the form of a whole unit, providing complete and lossless original scalar data support for real-time scalar condition verification and pre-pruning during subsequent index traversal.

[0032] Step 203: Perform structured encapsulation and matching of row identifiers, complete vector data, and inline scalar blocks, and complete field concatenation according to a unified storage format to obtain index tuples.

[0033] In this step, after reading the row identifier, complete vector data, and encapsulated inline scalar block corresponding to the current data row, a one-to-one structured association matching is performed on the three types of data entities to establish a logical binding relationship between the physical location identifier, vector semantic data, and scalar attribute data corresponding to the same data row. Following the underlying storage specifications of PostgreSQL index tuples and the arrangement rules of custom fused indexes, the row identifier, complete vector data, and inline scalar block are sequentially ordered and format adapted to form a unified storage unit with a fixed internal arrangement structure. Through standardized encapsulation and regular concatenation, a fused index tuple that simultaneously carries row location information, vector dimension information, and embedded scalar attribute information is generated. This index tuple can be directly used as the basic node unit of a hierarchical navigable small-world graph structure, providing a compliant and complete structured data carrier for subsequent overall index construction, graph node traversal, and real-time scalar filtering and pruning in the mixed query process.

[0034] Step 30: Based on the vector data of each index tuple, use the index tuple as nodes to construct a hierarchical navigable small world graph structure.

[0035] As an optional implementation, based on the vector data of each index tuple, the index tuple is used as a node to construct a hierarchical navigable small world graph structure. Specifically, it may include: Step 301: Generate the level to which the current index tuple to be inserted belongs, and use the level to which it belongs as the highest level number in which the current index tuple to be inserted appears in the hierarchical navigable small world graph structure.

[0036] In this step, for each index tuple to be inserted, a unique level is dynamically assigned to it based on the hierarchical generation rules of the hierarchical navigable small world graph using a probability distribution mechanism. This level serves as the upper limit of the current index tuple's level in the overall graph topology, limiting the highest level at which the index tuple can participate in topological connections. This ensures that the tuple is only eligible to exist as a node and be associated with neighbors in this level and all subordinate levels below it. By independently generating a unique level for each index tuple and labeling it as the highest level number, and following the top-down hierarchical topology construction logic of the hierarchical navigable small world graph, a hierarchical architecture foundation is laid for subsequent retrieval of nearest neighbor nodes from the highest global level downwards and the establishment of cross-level neighbor relationships. This ensures the randomness of the hierarchical distribution of the overall graph structure and the navigation characteristics of the retrieval topology.

[0037] Step 302: Starting from the current highest level of the hierarchical navigable small world graph structure, search downwards layer by layer, and use the vector distance metric to find the node that is closest to the vector data of the current index tuple to be inserted from the existing nodes of the current highest level.

[0038] In this step, after completing the hierarchical allocation and highest-level labeling of the current index tuple to be inserted, the search starts from the highest global level currently constructed in the graph structure. Node exploration proceeds layer by layer downwards according to a top-down hierarchical progression rule. During the search at each level, the correlation between the vector data of the current index tuple to be inserted and the vector data of all existing nodes in that level is calculated based on a preset vector space metric rule. Node similarity is evaluated using vector distance metric, and based on the evaluation results, the nearest neighbor node with the highest matching degree to the vector data of the index tuple to be inserted is selected from the existing nodes in the current level. This node serves as the benchmark node for continuing the search to the next level, until the target level corresponding to the index tuple to be inserted is gradually approached. This provides accurate nearest neighbor location basis for subsequent node insertion and neighbor relationship establishment.

[0039] Step 303: When the search reaches the corresponding level, insert the current index tuple to be inserted as a new node into the corresponding level and all levels below it, and establish a bidirectional neighbor connection between the new node and the nearest node in each level.

[0040] In this step, when the top-down hierarchical retrieval process reaches the level to which the current index tuple to be inserted belongs, it stops traversing downwards and uses that level as the starting point. The current index tuple to be inserted is treated as a new graph node and is sequentially placed into its own level and all lower-level topologies. Within each level that includes a new node, a bidirectional neighbor relationship is established between the new node and the nearest neighbor node, based on the closest node selected by the vector distance metric in the previous step. This makes them topological neighbors in the adjacency list of the corresponding level. By simultaneously completing node mounting and bidirectional neighbor link construction at multiple levels, the topological association structure of the hierarchical navigable small world graph across and within the same level is improved. This ensures that subsequent vector retrieval can jump and traverse between levels, and at the same time, it provides complete topological support for node exploration and scalar pre-pruning based on the graph structure during the query phase.

[0041] Step 304: Based on the preset upper limit of the number of neighbor connections, prune the neighbor list of new nodes in each level and retain the nearest neighbor node to complete the construction of the hierarchical navigable small world graph structure.

[0042] As an optional embodiment, based on a preset upper limit for the number of neighbor connections, the neighbor list of the new node in each level is pruned to retain the nearest neighbor node. Specifically, it may include: Step 3041: Traverse all the neighbor association nodes that the new node has established in the corresponding level, and calculate the vector space distance between the new node and each neighbor association node one by one.

[0043] In this step, after the new node completes the insertion of the corresponding level and the establishment of bidirectional neighbor connections, a complete traversal operation is performed on all neighbor nodes that have formed topological associations within the current level, visiting each bound neighbor node in turn. During the traversal, based on the inherent metric rules of the vector space, the vector data carried by the new node itself and the vector data corresponding to each neighbor node are independently calculated, and the distance between the two in the vector space is calculated accurately one by one. Through successive and item-by-item distance calculation operations, the spatial association metric results between the new node and all neighbor nodes within the current level are completely obtained, providing an objective and unified calculation basis for subsequent neighbor node sorting, filtering and topology pruning.

[0044] Step 3042: Sort all neighboring nodes in ascending order of vector space distance, and according to the preset neighbor connection number constraint, select the top neighboring nodes after sorting and keep them in the neighbor list.

[0045] In this step, after calculating the vector space distance between the new node and each of its neighboring nodes, the vector space distance is used as the sorting criterion to arrange all neighboring nodes at the current level in ascending order, so that nodes with higher spatial similarity are placed at the beginning of the sequence. Based on the pre-defined neighbor connection number constraint rules of the hierarchical navigable small-world graph structure, the sorted neighboring node sequence is truncated, retaining the leading neighboring nodes and including them in the new node's hierarchical neighbor list. This maintains the predetermined topological density and retrieval efficiency of the index graph structure, while avoiding memory overhead and retrieval path redundancy caused by an excessive number of neighboring nodes.

[0046] Step 3043: Remove remote neighbor relationships that exceed the quantity constraint, and synchronously update the reverse association topology on the side of the removed neighbors to complete the filtering and pruning of the new node neighbor list at all levels.

[0047] In this step, after sorting neighbor nodes by vector space distance and retaining the top-ranking nodes, the distant neighbor relationships exceeding the preset number constraint in the sorted sequence are removed and disconnected, breaking the established one-way and two-way topological links between the new node and such nodes. Simultaneously, the internal adjacency lists of each removed neighbor node are traversed, and the association record of the current new node is removed from its neighbor association topology, completing the synchronous unbinding and consistency maintenance of the two-way topological relationship. The distant neighbor removal and reverse topology update operations are performed layer by layer for each level where the new node is located, uniformly completing the normalization filtering and topology pruning of the neighbor list of the new node at all levels, so that the neighbor structure of each level of nodes maintains a regular and controllable topological form, ensuring the overall traversal efficiency of the hierarchical navigable small world graph and the stability of the index storage structure.

[0048] Step 40: Receive the mixed query command and traverse the node with the entry node of the hierarchical navigable small world graph structure as the current node to be visited.

[0049] As an optional embodiment, receiving a mixed query command and traversing the node with the entry node of the hierarchical navigable small world graph structure as the current node to be visited can specifically include: Step 401: Parsing the mixed query command to extract the query vector, scalar filtering predicate, and limiting the number of returned row identifiers.

[0050] In this step, upon receiving the user's vector-scalar hybrid query command, the entire query statement is deconstructed and semantically broken down using the SQL syntax parsing and semantic analysis mechanism of the PostgreSQL kernel. By identifying the vector similarity ranking syntax features in the query statement, query vector data for nearest neighbor retrieval is separated and extracted. Simultaneously, the conditional filtering syntax paragraphs in the statement are located, and the complete scalar filtering predicate logic is parsed and restored, preserving the original semantics, field relationships, and operational constraint rules of various scalar conditions. Furthermore, the syntax markers for limiting the number of query results are identified, and the limiting rules used to constrain the size of the final returned row identifiers are parsed and obtained. Through comprehensive semantic parsing and element decomposition, vector retrieval elements, scalar filtering elements, and result quantity constraint elements are completely separated from the hybrid query command, providing a complete query input basis for subsequent graph structure traversal of the fused index, node scalar condition verification, and determination of the convergence stopping condition of the result candidate set.

[0051] Step 402: Initialize an empty queue to be explored and an empty result candidate heap, and add the global entry node of the hierarchical navigable small world graph structure to the queue to be explored.

[0052] In this step, after parsing the mixed query command and obtaining the various query elements, the pre-query initialization operation is first performed. An empty queue to be explored and a result candidate heap are created, with these two data structures responsible for node traversal scheduling and temporary storage of compliant results, respectively. Then, the pre-defined global entry node of the hierarchical navigable small-world graph structure is located and placed into the initialized queue to be explored, serving as the starting access object for the entire graph node traversal process. By completing the initialization of the queue and candidate heap and the enqueue operation of the starting node, the basic scheduling framework for the mixed query phase is established, providing a standardized data flow carrier for subsequent layer-by-layer node exploration, scalar condition verification, and valid row identifier collection.

[0053] Step 403: Add a node to the queue to be explored as the current node to be visited, and traverse the hierarchical navigable small world graph structure.

[0054] In this step, node data in the queue to be explored is continuously read, and nodes are selected as the current nodes to be visited, thereby initiating the traversal process of the hierarchical navigable small-world graph structure. Relying on the inherent hierarchical topology and neighbor relationships of the graph structure, nodes are visited sequentially along the topological links extending from the current node to be visited, and the node exploration work is promoted in an orderly manner throughout the entire graph. This traversal process relies on the queue to be explored to realize the cyclic scheduling of nodes, ensuring that nodes at each level in the graph structure can be visited sequentially according to the predetermined topological logic, thereby providing a traversal basis for subsequent query actions such as reading the inline scalar blocks embedded in the nodes and performing scalar condition checks.

[0055] Step 50: Read the inline scalar block of the current node to be accessed, and store the row identifiers of the nodes that meet the preset scalar conditions into the result candidate heap. Stop traversing when the result candidate heap contains a preset number of row identifiers to complete the mixed query command.

[0056] As an optional embodiment, the inline scalar block of the currently accessed node is read, and the row identifiers of nodes that meet the preset scalar conditions are stored in the result candidate heap. When the result candidate heap contains a preset number of row identifiers, the traversal stops to complete the mixed query command. Specifically, it may include: Step 501: Read the original scalar value stored in the inline scalar block of the currently accessed node, and substitute the original scalar value into the scalar filter predicate for evaluation to obtain the Boolean judgment result.

[0057] In this step, for the graph node currently in the access state, the inline scalar block encapsulated inside its index tuple is read, and the complete and retained original scalar values ​​of various types are extracted. The extracted original scalar values ​​are substituted one by one into the scalar filtering predicate obtained in the previous parsing, and logical evaluation is performed according to the logical rules and operation relationships defined by the predicate, finally generating the corresponding Boolean judgment result. This result is used to determine whether the data record associated with the current node meets the scalar constraint conditions set by the query, providing a direct judgment basis for subsequent node filtering, result collection, and traversal path pruning.

[0058] Step 502: If the Boolean judgment result is false, then discard the currently visited node.

[0059] In this step, if the Boolean judgment result obtained by logical evaluation is false, it indicates that the scalar data corresponding to the node to be accessed cannot match the filtering conditions set by the query. Then, the node is discarded and no further operations such as neighbor node traversal, vector comparison, and result aggregation are performed for this node. Through this immediate discarding logic, path pruning can be completed during the index traversal process, avoiding invalid nodes from continuing to occupy computing and scheduling resources, simplifying the overall traversal range, and improving the overall running efficiency of the hybrid query process.

[0060] Step 503: If the Boolean judgment result is true, then read the inline storage vector data from the currently accessed node, and calculate the distance between the inline storage vector data and the query vector in the mixed query command to obtain the distance value.

[0061] In this step, if the Boolean judgment result obtained by logical evaluation is true, it means that the node to be accessed meets the scalar constraint conditions corresponding to the query. Then, the complete vector data embedded in the index tuple of the node is read. According to the vector space measurement rules, the vector data is compared with the query vector extracted from the mixed query command to generate the corresponding vector distance value. This distance value can reflect the similarity between the two sets of vectors in space, providing the core evaluation basis for subsequent screening of neighboring nodes and collection of effective query results.

[0062] Step 504: Based on the distance value, insert the row identifier of the current node to be visited into the result candidate heap, and add all the neighbor nodes of the current node to be visited to the exploration queue, until the exploration queue is empty and / or the result candidate heap contains a limited number of row identifiers, then stop traversing.

[0063] As an optional embodiment, the row identifier of the currently visited node is inserted into the result candidate heap based on the distance value. Specifically, it may include: Step 5041: If the result candidate heap is not full, the row identifier of the currently visited node is directly inserted into the result candidate heap.

[0064] In this step, after calculating the distance between the current node vector data and the query vector and obtaining the corresponding distance value, the internal storage status of the result candidate heap is checked to determine whether the current capacity of the data structure has reached saturation. When it is confirmed that the result candidate heap still has remaining storage space and has not yet been filled with data, the unique row identifier bound to the currently accessed node is extracted, and the data insertion operation is performed to formally store the row identifier into the result candidate heap. During the entire insertion process, the result candidate heap will rely on its own preset organization rules and combine the calculated vector distance value to complete the internal sorting and arrangement, so that the stored row identifiers are arranged in an orderly manner according to the vector similarity feature. This processing method can collect valid data identifiers that simultaneously meet the scalar filtering conditions and vector retrieval requirements in real time during the graph structure traversal, continuously accumulate compliant query results, and dynamically manage the results by leveraging the characteristics of the candidate heap. This ensures that records that meet the conditions can be retained in a timely manner and maintains the orderliness of the result set, building a complete data buffer system for subsequent determination of traversal termination conditions, organization and output of final query results, and ensuring the smooth and orderly progress of the hybrid query process.

[0065] Step 5042: If the result candidate heap is full and the distance value is less than the maximum distance value in the heap, then replace the row identifier with the row identifier of the current node with the row identifier of the heap with the maximum distance.

[0066] In this step, when the candidate result stack is found to be full, the maximum distance value of the records in the stack is retrieved as the comparison benchmark. The vector distance value calculated by the current node is compared with it. If the distance value corresponding to the current node is smaller, it means that the node has a higher similarity to the query vector in the vector space. At this time, a replacement operation is performed, the row identifier with the largest vector distance in the candidate stack is taken out and removed, and the row identifier of the current compliant node is stored in the corresponding position. The candidate stack will simultaneously readjust the internal element arrangement according to the vector distance sorting rules to continuously maintain the orderliness of the stack structure. This mechanism can dynamically filter the query results with better overall matching degree under the premise of fixed storage space, continuously optimize the overall quality of the candidate set, ensure that the remaining row identifier always corresponds to the data record with the highest vector similarity, realize the optimal update of query results within the limited storage range, and ensure the accuracy of the final output result of the hybrid query.

[0067] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a database-based fusion index construction and query device, the structure of which is as follows: Figure 2 As shown.

[0068] Figure 2 This is a schematic diagram of the internal structure of a database-based fusion index construction and query device provided in an embodiment of this application. For example... Figure 2 As shown, the device includes: At least one processor 201; And a memory 202 that is communicatively connected to at least one processor; The memory 202 stores instructions executable by at least one processor. These instructions are executed by at least one processor 201 to enable the processor 201 to: receive an index creation command and parse vector column names and scalar column names from the command; read vector data corresponding to the vector column names, original scalar values ​​corresponding to the scalar column names, and row identifiers from a preset database table, and combine the vector data, original scalar values, and row identifiers into index tuples; construct a hierarchical navigable small-world graph structure by using the index tuples as nodes based on the vector data of each index tuple; receive a mixed query command and traverse the nodes by using the entry node of the hierarchical navigable small-world graph structure as the current node to be visited; read the inline scalar blocks of the current node to be visited and store the row identifiers of nodes that meet preset scalar conditions into a result candidate heap, stopping traversal when the result candidate heap contains a preset number of row identifiers, thus completing the mixed query command.

[0069] Some embodiments of this application provide corresponding to Figure 1A non-volatile computer storage medium for database-based fusion index construction and querying stores computer-executable instructions. These instructions are configured to: receive an index creation command and parse vector column names and scalar column names from the command; read vector data corresponding to the vector column names, original scalar values ​​corresponding to the scalar column names, and row identifiers from a preset database table, and combine these into index tuples; construct a hierarchical navigable small-world graph structure using the vector data of each index tuple as nodes; receive a fusion query command and traverse nodes using the entry node of the hierarchical navigable small-world graph structure as the current node to be accessed; read the inline scalar blocks of the currently accessed node and store the row identifiers of nodes that meet preset scalar conditions into a result candidate heap; stop traversing when the result candidate heap contains a preset number of row identifiers, thus completing the fusion query command.

[0070] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0071] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0072] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0077] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0078] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0079] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A database-based fusion index construction and query method, characterized in that, The method includes: Receive an index creation command and parse the vector column names and scalar column names from the index creation command; Read the vector data corresponding to the vector column name, the original scalar value corresponding to the scalar column name, and the row identifier from the preset database table, and combine the vector data, the original scalar value, and the row identifier into an index tuple; Based on the vector data of each index tuple, the index tuple is used as a node to construct a hierarchical navigable small world graph structure. Receive mixed query commands and perform node traversal using the entry node of the hierarchical navigable small world graph structure as the current node to be visited. Read the inline scalar block of the currently accessed node, and store the row identifiers of nodes that meet the preset scalar conditions into the result candidate heap. Stop traversing when the result candidate heap contains a preset number of row identifiers to complete the mixed query command. Based on the vector data of each index tuple, the index tuples are used as nodes to construct a hierarchical, navigable small-world graph structure, specifically including: Generate the hierarchy to which the current index tuple to be inserted belongs, and use the hierarchy as the highest level number in which the current index tuple to be inserted appears in the hierarchical navigable small world graph structure; Starting from the current highest level of the hierarchical navigable small world graph structure, search downwards layer by layer, and use the vector distance metric to find the node that is closest to the vector data of the current index tuple to be inserted from the existing nodes of the current highest level. When the search reaches the level to which it belongs, the current index tuple to be inserted is inserted as a new node into the level to which it belongs and all levels below it, and a bidirectional neighbor connection is established between the new node and the nearest node in each level. Based on a preset upper limit for the number of neighbor connections, the neighbor list of the new node in each level is pruned, retaining the nearest neighbor node to complete the construction of a hierarchical navigable small world graph structure.

2. The database-based fusion index construction and query method according to claim 1, characterized in that, Receive an index creation command and parse the vector column names and scalar column names from the index creation command, specifically including: The syntax structure of the index creation command is broken down to separate the main index definition part from the additional include clause part; Extract the vector column identifier information used to construct the vector retrieval dimension from the main part of the index definition, and extract the scalar column identifier information of the index node from the additional containing clause part; Filter out duplicate column identifier information and organize the vector column name sequence and the scalar column name sequence according to the syntax definition order.

3. The database-based fusion index construction and query method according to claim 1, characterized in that, The system reads vector data corresponding to the vector column name, the original scalar value corresponding to the scalar column name, and the row identifier from a preset database table, and combines the vector data, the original scalar value, and the row identifier into an index tuple, specifically including: Traverse each data row in the preset database table and obtain the row identifier of the current data row; Based on the vector column names and the scalar column names, read the complete vector data and the original field values ​​of each scalar column from the current data row, and encapsulate the original field values ​​of each scalar column into an inline scalar block; The row identifier, the complete vector data, and the inline scalar block are structured and encapsulated for matching, and the fields are concatenated according to a unified storage format to obtain the index tuple.

4. The database-based fusion index construction and query method according to claim 1, characterized in that, Based on a preset upper limit for the number of neighbor connections, the neighbor list of the new node in each level is pruned, retaining the nearest neighbor node. Specifically, this includes: Traverse all neighboring nodes that the new node has established at the corresponding level, and calculate the vector space distance between the new node and each neighboring node one by one; Sort all neighboring nodes in ascending order of distance in the vector space, and select the top neighboring nodes after sorting according to the preset neighbor connection number constraint and keep them in the neighbor list. Remove distant neighbor relationships that exceed the quantity constraints and simultaneously update the reverse association topology on the side of the removed neighbors to complete the filtering and pruning of the new node neighbor list at all levels.

5. The database-based fusion index construction and query method according to claim 1, characterized in that, Receiving a mixed query command and traversing the node as the current node to be visited, specifically including: The mixed query command is parsed to extract the query vector, scalar filter predicate, and limit the number of returned row identifiers; Initialize an empty queue to be explored and an empty result candidate heap, and add the global entry node of the hierarchical navigable small world graph structure to the queue to be explored; Add a node to the queue to be explored as the current node to be visited, and traverse the hierarchical navigable small world graph structure.

6. The database-based fusion index construction and query method according to claim 5, characterized in that, Read the inline scalar block of the currently accessed node, and store the row identifiers of nodes that meet the preset scalar conditions into the result candidate heap. Stop traversing when the result candidate heap contains a preset number of row identifiers to complete the mixed query command, specifically including: Read the original scalar value stored in the inline scalar block from the current node to be accessed, and substitute the original scalar value into the scalar filtering predicate for evaluation to obtain the Boolean judgment result; If the Boolean judgment result is false, then the currently accessed node is discarded; If the Boolean judgment result is true, then the inline stored vector data is read from the currently accessed node, and the distance between the inline stored vector data and the query vector in the mixed query command is calculated to obtain the distance value; Based on the distance value, the row identifier of the currently visited node is inserted into the result candidate heap, and all neighbor nodes of the currently visited node are added to the exploration queue, until the exploration queue is empty and / or the result candidate heap contains the limited number of row identifiers, at which point traversal stops.

7. The database-based fusion index construction and query method according to claim 6, characterized in that, Based on the distance value, the row identifier of the currently visited node is inserted into the result candidate heap, specifically including: If the result candidate stack is not full, the row identifier of the currently accessed node is directly inserted into the result candidate stack; If the candidate heap is full and the distance value is less than the maximum distance value in the heap, then the row identifier with the largest distance in the heap is replaced with the row identifier of the current node.

8. A database-based fusion index building and querying device, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform a method as described in any one of claims 1-7.

9. A non-volatile computer storage medium for constructing and querying fused indexes based on a database, storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they implement a method as described in any one of claims 1-7.

Images