Picture index construction method and device, equipment, storage medium and program product

Abstract

The application provides a graph index construction method and device, equipment, storage medium and program product, and relates to the fields of data retrieval and vector retrieval. The method comprises the following steps: obtaining an existing neighbor set and a new candidate set of a target node; the existing neighbor set is a neighbor node set existing in a graph index of the target node; the new candidate set is a to-be-screened neighbor node set generated by local search or random sampling; performing merging processing on the existing neighbor set and the new candidate set to generate an initial candidate set; sorting the distances between each candidate node in the initial candidate set and the target node to obtain a sorted candidate set; and based on the sorted candidate set, truncating candidate nodes with a preset distance to generate a target candidate set. The method is used to reduce the candidate processing range, reduce redundant calculation and sorting burden, improve the graph index construction and updating efficiency, and improve the graph structure connectivity and retrieval performance.

Description

Technical Field

[0001] This application relates to the fields of data retrieval and vector retrieval, and in particular to a graph index construction method, apparatus, device, storage medium, and program product. Background Technology

[0002] In large-scale high-dimensional vector retrieval, existing technologies generally adopt a graph-based approximate nearest neighbor index construction and query method.

[0003] However, existing graph index construction often relies on neighbor selection methods that truncate the nodes after sorting by distance from the target node. In high-dimensional and large-scale scenarios, this can easily lead to localized concentration of neighbor distribution, affecting graph connectivity and retrieval efficiency. At the same time, when the candidate set is large, the overhead of distance calculation, state initialization, and multiple rounds of traversal is high, which limits the construction efficiency.

[0004] Therefore, how to improve the efficiency of graph index construction and reduce redundant computational overhead while ensuring the quality of graph indexes has become a technical problem that needs to be solved. Summary of the Invention

[0005] This application provides a graph index construction method, apparatus, device, storage medium, and program product to solve the problems of local concentration in the selection of target node neighbors and the difficulty in balancing graph structure quality and construction efficiency during the graph index construction process, thereby improving the processing efficiency of the graph index construction process while ensuring controllable graph index quality.

[0006] In a first aspect, embodiments of this application provide a graph index construction method, the method comprising:

[0007] Obtain the existing neighbor set and the new candidate set of the target node; the existing neighbor set is the set of neighbor nodes that already exist in the graph index of the target node; the new candidate set is the set of neighbor nodes to be filtered generated through local search or random sampling.

[0008] The existing neighbor set and the newly added candidate set are merged to generate an initial candidate set;

[0009] The candidate nodes in the initial candidate set are sorted according to their distances to the target node to obtain the sorted candidate set;

[0010] Based on the sorted candidate set, candidate nodes with a preset distance are truncated to generate the target candidate set.

[0011] In one possible embodiment, the process of merging the existing neighbor set and the newly added candidate set includes:

[0012] Eliminate duplicate nodes in the existing neighbor set and the new candidate set using set operations;

[0013] Remove self-linked nodes from the existing neighbor set and the newly added candidate set based on conditional judgment; self-linked nodes are nodes that are the same as the target node.

[0014] In one possible embodiment, the existing neighbor set and the newly added candidate set are merged to generate an initial candidate set, which includes at least one of the following:

[0015] The initial candidate set size is dynamically adjusted based on the vector distribution density of the target nodes; the vector distribution density is the density of node distribution in the vector space.

[0016] The initial candidate set size is dynamically adjusted based on the real-time system load.

[0017] In one possible embodiment, the candidate nodes in the initial candidate set are sorted according to their distances to the target node to obtain a sorted candidate set, including:

[0018] Calculate the vector distance between each candidate node in the initial candidate set and the target node; the vector distance is the numerical value of the similarity between two vectors.

[0019] The candidate nodes in the initial candidate set are sorted in descending order based on vector distance to obtain the sorted candidate set.

[0020] In one possible embodiment, the vector distance between each candidate node in the initial candidate set and the target node is calculated, including at least one of the following:

[0021] Calculate the distance based on the topological connectivity between the candidate node and the target node;

[0022] The distance is calculated based on the semantic feature labels of the candidate node and the target node.

[0023] In one possible embodiment, before sorting the candidate nodes in the initial candidate set in descending order based on vector distance to obtain the sorted candidate set, the process includes:

[0024] Load the node vectors from the existing neighbor set and perform distance calculation and state initialization;

[0025] The distance calculation state is initialized and cached.

[0026] In one possible embodiment, after generating the target candidate set by truncating candidate nodes with a preset distance based on the sorted candidate set, the process includes:

[0027] Calculate the vector distance between nodes based on the node vectors in the target candidate set;

[0028] The occlusion status of subsequent candidate nodes is dynamically updated based on the vector distance. The occlusion status is a status marker indicating whether a candidate node is covered by existing neighbor nodes.

[0029] In one possible embodiment, the vector distance between nodes is calculated based on the node vectors in the target candidate set, including:

[0030] The vector distance between nodes is calculated in parallel using hardware parallelization technology; hardware parallelization technology utilizes the parallel computing capabilities of graphics processing units or field-programmable gate arrays to process multiple computing tasks simultaneously.

[0031] Reduce data transfer overhead for vector distance calculations through memory optimization techniques.

[0032] Secondly, embodiments of this application provide a graph index construction apparatus, the apparatus comprising:

[0033] The acquisition module is used to acquire the existing neighbor set and the new candidate set of the target node; the existing neighbor set is the set of neighbor nodes that already exist in the graph index of the target node; the new candidate set is the set of neighbor nodes to be screened generated through local search or random sampling.

[0034] The merging module is used to merge the existing neighbor set and the newly added candidate set to generate an initial candidate set;

[0035] The sorting module is used to sort the candidate nodes in the initial candidate set according to the distance between each candidate node and the target node, so as to obtain a sorted candidate set;

[0036] The generation module is used to truncate candidate nodes based on the sorted candidate set, retaining a preset distance, and generate the target candidate set.

[0037] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0038] The memory stores computer-executed instructions;

[0039] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in any of the first aspects above.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0041] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0042] This application provides a graph index construction method, apparatus, device, storage medium, and program product. By obtaining the existing neighbor set and the newly added candidate set of the target node, merging the two to generate an initial candidate set, and sorting the candidate nodes in the initial candidate set according to the distance between each candidate node and the target node, the candidate nodes are truncated based on the sorted candidate set and retained at a preset distance. This can effectively shrink the candidate range while retaining candidate nodes with strong correlation to the target node, reducing redundant calculations and traversal overhead in subsequent graph index construction, thereby improving the efficiency of graph index construction while taking into account the quality of the graph index. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 A flowchart illustrating a graph index construction method provided in this application;

[0045] Figure 2 A flowchart illustrating a graph index neighbor pruning method provided in this application;

[0046] Figure 3 A schematic diagram of a graph index construction device provided in this application;

[0047] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0051] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation access points for users to choose to authorize or refuse.

[0052] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0053] First, let me explain the terms used in this application:

[0054] Graphics Processing Unit (GPU): A coprocessor used for processing images and graphics operations.

[0055] Field-Programmable Gate Array (FPGA): refers to a programmable logic device whose core architecture consists of programmable logic blocks, input / output modules, programmable interconnect resources, and static memory.

[0056] Large-scale high-dimensional vector retrieval belongs to the field of data retrieval and approximate nearest neighbor index construction technology, and is commonly used in scenarios such as e-commerce recommendation, image retrieval, and multimodal retrieval. In these scenarios, the system typically works collaboratively based on a vector database, a graph index construction module, and an online query module to represent user, product, or image features as vector nodes, and organize the adjacency relationships between nodes through a graph structure.

[0057] Existing graph indexing methods typically generate candidate neighbors for the target node, then sort them according to the distance between the candidate nodes and the target node, and select the node with the best distance as the neighbor. Alternatively, candidate nodes can be selected based on local search results or random sampling results to participate in the establishment of neighbor connections in the graph structure.

[0058] However, this method is prone to problems such as excessively large candidate size and excessive distance calculation in high-dimensional and large-scale data conditions. Especially when existing neighbors and new candidates coexist, if there is no unified merging process, duplicate nodes and invalid nodes will enter the subsequent sorting process, causing additional computational overhead. On the other hand, simply relying on distance sorting and truncation can also easily cause the retained neighbor nodes to be too concentrated in local areas, which is not conducive to maintaining good connectivity of the graph structure.

[0059] Furthermore, when the number of candidate sets reaches a large scale, both the distance calculation before sorting and the filtering process after sorting significantly prolong the graph index construction time, thereby affecting the index update efficiency and the overall responsiveness of the retrieval system. Therefore, how to reduce candidate processing redundancy and improve graph index construction efficiency while ensuring the quality of neighbor selection has become an urgent technical problem to be solved in this field.

[0060] Considering the aforementioned problems with existing graph index construction methods, this application proposes a graph index construction method. After obtaining the existing neighbor set and the new candidate set of the target node, the two are first merged to generate an initial candidate set. Then, the candidate nodes in the initial candidate set are sorted according to their distances to the target node, resulting in a sorted candidate set. Finally, based on the sorted candidate set, candidate nodes with a preset distance are truncated to generate the target candidate set. This technical approach allows for pre-constraint of the processing range during the candidate selection stage, providing a foundation for improving graph index construction efficiency and reducing redundant computational overhead.

[0061] The entity executing the index construction method can be, for example, a construction system. Optionally, the construction system can be any existing electronic device with processing capabilities, such as a terminal or a server. In some embodiments, the construction system can also be deployed in a server cluster or cloud environment. This application does not limit the deployment environment of the construction system.

[0062] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0063] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0064] Figure 1 This is a flowchart illustrating a graph index construction method provided in this application. Figure 1 As shown, the method includes:

[0065] S101, obtain the existing neighbor set and the new candidate set of the target node; the existing neighbor set is the set of neighbor nodes that already exist in the graph index of the target node; the new candidate set is the set of neighbor nodes to be screened generated through local search or random sampling.

[0066] For example, the target node can be the central node in the graph index currently performing neighbor update or neighbor reselection processing, and subsequent candidate merging, distance calculation, sorting, and truncation all revolve around this target node.

[0067] For example, the existing neighbor set can be the set of neighbor nodes with which the target node has already established connections in the current graph index structure. For instance, the existing neighbor set can be stored in the adjacency list, adjacency array, or a memory-mapped structure organized based on node identifiers corresponding to the graph index.

[0068] For example, the new candidate set can be a set of nodes to be filtered in the current round to supplement the range of neighbor selections. This set can come from local search results or from random sampling results.

[0069] For example, a local search can be performed on the existing structure of the graph index, starting from the target node or an entry node related to the target node, to obtain a batch of candidate nodes that are reachable from the target node in the current graph structure.

[0070] For example, random sampling can be performed by selecting several nodes from the full set of nodes or the set of partitioned nodes according to a preset sampling rule to form the input to be filtered.

[0071] Optionally, the construction system can be a graph index construction module in a vector database, an offline index update service, or an online incremental graph construction service. For example, the construction system can receive the identifier of the target node to be processed, and then read the existing set of neighbors corresponding to the target node from the graph index storage structure based on the identifier, and receive the new candidate set from the candidate generation module.

[0072] Optionally, the candidate generation module can output a candidate list based on local search, a candidate list based on random sampling, or a combination of both types of results as input for a new candidate set. If the new candidate set comes from different sources, the candidate generation module can retain the source labels.

[0073] Optionally, the system can store target nodes and candidate nodes in the form of a dimensional vector. Each node corresponds to a unique node identifier and related data address, and both the existing neighbor set and the new candidate set must contain at least the node identifier. When processing a target node, the system can directly read this set based on the target node's node identifier.

[0074] Optionally, when obtaining an existing set of neighbors, the system can read the current number of neighbors of the target node, the adjacency information, and the node validity flag; when obtaining a new set of candidates, the system can simultaneously verify whether the candidate nodes have been deleted, whether they belong to the current graph construction range, and whether they have complete data representation.

[0075] Optionally, for candidate nodes generated by local searches, the system can record the search source. For candidate nodes generated by random sampling, the system can directly generate a fixed-length candidate list based on a preset sampling size, or determine the sampling quantity proportionally based on the current index size. If the graph index uses sharding or partitioning storage, for example, the system can read the existing neighbor set from the shard where the target node is located, and form a new candidate set from search results within the same shard, search results across shards, or global sampling results, and aggregate them into a unified cache.

[0076] Based on the above analysis, by simultaneously obtaining the existing neighbor set and the newly added candidate set of the target node within the same processing round, both already connected nodes and newly introduced candidate nodes enter a unified candidate processing flow, thus providing a complete input range for subsequent set merging and distance filtering. The two types of nodes do not need to be independently sorted, and the candidate processing path remains consistent, reducing the computational overhead of multiple rounds of repeated filtering. It should be understood that the above example is merely illustrative and not limiting.

[0077] S102, merge the existing neighbor set and the newly added candidate set to generate the initial candidate set.

[0078] For example, the initial candidate set can be an intermediate set that is a unified summary of the existing neighbor set and the new candidate set, which is used to carry out subsequent distance calculation and sorting processing.

[0079] For example, the merging process could be to uniformly organize the two types of candidate inputs.

[0080] Optionally, the system can merge the existing neighbor set and the newly added candidate set according to a unified rule, and output an initial candidate set. If the node record carries source information, the system can integrate the relevant source information during the merging process, and then export the retained nodes as the initial candidate set according to a preset storage order.

[0081] Optionally, the system can use an array of node identifiers as input for the existing neighbor set and the new candidate set, and can use appropriate data structures or processing methods to complete the set merging. The system can also concatenate the existing neighbor set and the new candidate set, and then form an initial candidate set according to preset rules.

[0082] Optionally, the system can store the initial candidate set in a contiguous memory buffer for subsequent batch reading of relevant data and distance calculation. For large-scale candidate inputs, the system can perform block-based merging processing. For example, the system can first process the existing neighbor set, then receive new candidate sets block by block, and finally output the initial candidate set after all blocks have been processed.

[0083] Optionally, the system can first read the local neighbor set of the computing node where the target node is located, and then receive the remote candidate shards, and perform merging according to unified rules to obtain the initial candidate set for this round.

[0084] As one possible implementation, the system can eliminate duplicate nodes in the existing neighbor set and the new candidate set through set operations, and remove self-linked nodes in the existing neighbor set and the new candidate set through conditional judgment, where a self-linked node is a node that is the same as the target node.

[0085] Optionally, the system can read the existing neighbor set and the newly added candidate set corresponding to the target node, and map the nodes in both sets to a unified set of node identifiers. Then, the node identifiers are input into the duplicate node elimination module to complete the duplicate node elimination. The system can compare the node identifiers after duplicate node elimination with the target node identifiers. If they match, the node is determined as a self-removed node and removed from the initial candidate set. If they do not match, the node is retained as a valid candidate node. After the above processing, the output initial candidate set only contains valid nodes after duplicate node elimination and does not contain invalid nodes that are the same as the target node.

[0086] By using the above method, after set operations and conditional judgment, the merged candidate set can maintain the uniqueness and validity of nodes, reduce duplicate nodes and self-node nodes from entering the subsequent distance sorting process, thereby reducing the amount of invalid distance calculation, making the subsequent neighbor selection results more stable, reducing redundant processing overhead and improving index building efficiency.

[0087] As one possible implementation, the system can dynamically adjust the size of the initial candidate set based on the vector distribution density of the target nodes, and then dynamically adjust the size of the initial candidate set based on the real-time system load. Vector distribution density refers to the density of node distribution in the vector space.

[0088] Optionally, the system can generate an initial candidate set after merging based on a basic size, and then expand or shrink this basic size according to the vector distribution density of the target nodes. For example, when the vector distribution density is high, the system can expand the initial candidate set size to a preset upper limit to retain more local adjacency relationships; when the vector distribution density is low, the system can compress the initial candidate set size to a preset lower limit to reduce the number of invalid candidates.

[0089] Optionally, the system can also adjust the size of the initial candidate set according to the real-time load of the system through the candidate interception control unit. For example, the initial candidate set size can be reduced when the load is high and increased when the load is low, so that the size of the initial candidate set changes synchronously with the resource status.

[0090] In this way, the merged candidate nodes can be adaptively pruned or expanded according to the local vector distribution state or the system operating state, so that the initial candidate set can control the amount of computation while maintaining the necessary adjacency information, and provide a more stable input basis for subsequent distance sorting and truncation retention, thereby improving the efficiency of graph index construction and reducing redundant processing overhead.

[0091] S103, sort the candidate nodes in the initial candidate set according to the distance between each candidate node and the target node to obtain the sorted candidate set.

[0092] For example, the sorted candidate set can be a set obtained by arranging the candidate nodes in the initial candidate set in order according to their distances to the target node. The target node serves as a unified reference point for distance calculation, and each candidate node in the initial candidate set can have its corresponding distance calculated with the target node.

[0093] In some embodiments, such as when nodes are represented by high-dimensional vectors, the distance can be Euclidean distance, equivalent distance form corresponding to the inner product, cosine distance, or other distance rules preset by the system, as long as the proximity of the candidate node to the target node can be converted into a comparable value.

[0094] Optionally, the system can acquire the data corresponding to the target node, then traverse the initial candidate set, sequentially acquire the data corresponding to each candidate node, calculate the distance between the candidate node and the target node, and write the node identifier and the corresponding distance value to form a distance record and write it to the sorting cache. After completing all distance calculations, the system can sort the data according to a preset sorting rule to obtain a sorted candidate set.

[0095] Optionally, the system can perform distance calculations node by node or in batches. For example, if batch calculation is used, the system can organize the data corresponding to multiple candidate nodes, perform parallel operations with the data corresponding to the target node to generate a corresponding distance array, and then associate it with the candidate node identifier.

[0096] Optionally, the system can use quicksort, heapsort, mergesort, or partial sorting algorithms for sorting. If only candidate nodes within a preset range need to be retained, the system can use a heap or selection algorithm to first obtain the first few candidate nodes with better distances, and then perform a locally ordered arrangement based on these nodes to form a sorted candidate set for truncation processing. If multiple candidate nodes have the same distance value, the system can determine the output result based on the node identifier order, writing order, or source order.

[0097] Optionally, the system can use consistent data representation to define the rules and numerical precision of target and candidate nodes. For very large initial candidate sets, the system can employ batch computation and incremental merge sorting, where the distances to several candidate nodes are calculated in each batch to generate locally ordered results, and then multiple locally ordered results are merged to reduce one-time memory usage.

[0098] In one possible implementation, the system can calculate the vector distance between each candidate node in the initial candidate set and the target node, and then sort the candidate nodes in the initial candidate set in descending order based on the vector distance to obtain a sorted candidate set, where the vector distance is the numerical value of the similarity between two vectors.

[0099] Optionally, after calculating the vector distance, the system can sort the candidate nodes in the initial candidate set in descending order according to the numerical value of the vector distance, generating a sorted candidate set. This descending sort can be implemented by the sorting module based on the candidate records in memory, or the vector retrieval engine can directly output the sorting results after batch calculation.

[0100] Optionally, the system can receive the target node and its initial candidate set, perform vector distance calculation on each candidate node in the set, and use the resulting vector distance as the sorting key to perform descending sorting, thereby obtaining a sorted candidate set. This sorted candidate set retains candidate nodes whose vector distance values ​​better meet the preset screening requirements and provides an ordered basis for subsequent truncation, giving the candidate processing process a clear priority.

[0101] The above method, which uses a unified vector distance calculation and descending order sorting, can make the candidate node screening results consistent and reusable. It can reduce the processing overhead caused by unordered traversal while ensuring the interpretability of candidate sorting, and enable the subsequent truncation stage to directly complete the retention based on the sorting results, thereby improving the processing efficiency of graph index construction.

[0102] As one possible implementation, the system can calculate the distance based on the topological connectivity between the candidate node and the target node, and then calculate the distance based on the semantic feature labels of the candidate node and the target node.

[0103] Optionally, after generating the initial candidate set, the system can obtain the topological connectivity information corresponding to the target node for each candidate node. The topological connectivity information can be provided by the adjacency table, edge weight record table, or reachable path record table in the graph index. For example, if the vector distance calculation method is based on topological connectivity, the system can use the shortest path length, hop count, or path weight between the candidate node and the target node as the basic value of the distance, and combine it with a preset mapping rule to convert the basic value into a vector distance for sorting.

[0104] Optionally, if there is no direct path between a candidate node and a target node, the system can use the connectivity cost through intermediate nodes as a supplementary factor to ensure that the vector distance reflects the indirect connections between nodes. For semantic feature labeling, the system can obtain the label codes corresponding to both candidate and target nodes, calculate the semantic distance through label consistency, label hierarchy differences, or the intersection / union of label sets, and then map the semantic distance to a vector distance for unified ranking with other candidate nodes. The label codes can be stored in the metadata field of the vector database. In practical applications, other encoding formats can also be selected, and this application does not impose any limitations on this.

[0105] Optionally, the system can, based on the initial candidate set, call either the topological connectivity calculation module or the semantic feature label calculation module to obtain the vector distance results for each candidate node, and output these results as the basis for candidate ranking to subsequent ranking units, thereby forming a ranked candidate set with higher relevance to the target node. By incorporating graph structural connectivity or semantic attributes into the vector distance calculation process, the ranking results of candidate nodes can simultaneously reflect numerical similarity and structural relevance, making the final retained target candidate set closer to the true proximity relationships of the target node.

[0106] By combining the above methods with graph structure connectivity information or semantic label information to comprehensively characterize the vector distance of candidate nodes, we can improve the ability of the vector distance results to describe the relevance of candidates and make the candidate nodes that are subsequently truncated and retained more in line with the requirements of graph index construction, thereby improving the accuracy of neighbor selection and the quality of graph structure organization.

[0107] In some embodiments, before sorting the candidate nodes in the initial candidate set in descending order based on vector distance to obtain the sorted candidate set, the system can load the node vectors in the existing neighbor set and initialize the distance calculation state, and then cache the distance calculation state initialization.

[0108] Optionally, before the existing neighbor set corresponding to the target node enters the sorting stage, the system can load the neighbor node vectors in batches into the memory cache of the computing node based on the index building module, and initialize the distance calculation state according to the vector dimension, distance metric, and calculation precision requirements of the target node. This state may include the alignment information of the target node vector, component accumulation register, temporary difference vector, and context handle for batch calculation. The relevant state is written to the cache after initialization so that it can be directly called when traversing the initial candidate set.

[0109] Optionally, if distance calculation is performed on each candidate node in the initial candidate set, the distance calculation state in the cache can reuse the basic computing environment corresponding to the existing neighbor set. The system does not need to repeatedly read the node vectors in the existing neighbor set, nor does it need to rebuild the same computing context.

[0110] Optionally, if the node vectors in the existing neighbor set are represented using fixed-length floating-point numbers, the system can read them continuously according to a preset byte width and maintain a data layout consistent with the target node vector in the cache, thereby allowing subsequent distance calculations for different candidate nodes to share the same initialization result. In practical applications, this cache can be implemented using a circular buffer, a hash table, or a memory object pool; this application does not limit its implementation.

[0111] By adopting the above method, the node vector reading and distance calculation preparation of the existing neighbor set can be completed and reused once before sorting. The sorting process does not need to repeat the state construction when processing the initial candidate set, thereby reducing the initialization overhead and shortening the preparation time before sorting the candidate nodes, making the generation process of the sorted candidate set more stable.

[0112] S104: Based on the sorted candidate set, truncate and retain candidate nodes with a preset distance to generate the target candidate set.

[0113] For example, the target candidate set can be the final candidate set obtained by performing truncation and retention on the sorted candidate set, which will serve as the input for the target node to subsequently establish or update adjacency relationships.

[0114] For example, the preset distance can be a preset condition for defining the retention range. This condition can be directly expressed as a distance threshold, or as the upper limit of the quantity corresponding to the distance sorting result, or as a composite rule that simultaneously satisfies the distance threshold and the upper limit of the quantity.

[0115] Optionally, the system can sequentially traverse candidate nodes starting from the first position of the sorted candidate set. If the distance between a candidate node and the target node meets a preset distance requirement, the system can write the candidate node into the target candidate set. When a node that does not meet the preset distance requirement is encountered, the system can stop writing and output the written candidate nodes as the target candidate set. The preset distance requirement can be that the distance value between candidate nodes is not greater than a pre-set distance threshold.

[0116] Optionally, the system can retain only the top K candidate nodes from the sorted candidate set, using them as the target candidate set, by setting a preset distance as the upper limit K. Alternatively, the system can perform an initial screening based on a distance threshold, and then retain the top K nodes from those that meet the threshold, thus controlling both the candidate distance range and the size of the target candidate set.

[0117] For example, if the length of the candidate set for sorting is less than the upper limit K, the system can choose to include all candidate nodes in the target candidate set. If there are multiple nodes in the candidate set that are equidistant from the truncation boundary, the system can retain them according to the order in the sorting results up to the upper limit, or it can continue to retain all nodes that are equidistant from the boundary node and meet the threshold.

[0118] Optionally, for the distance threshold method, the threshold can be a fixed value or dynamically determined by the data distribution of the target node, such as being generated based on the distance statistics of the target nodes in the current batch. The key is to establish a clear and comparable preset distance condition during this step. For distributed scenarios, the sorting candidate set can first obtain a globally ordered result locally and then perform a unified truncation; alternatively, it can first merge the locally ordered results of each shard before truncation.

[0119] Optionally, the construction system can use a fixed-length array, a dynamic array, or a compact adjacency cache to store the target candidate set. After writing, the construction system can submit the target candidate set to the graph index update module to establish connections between the target node and each target candidate node, or to replace the original neighbor list of the target node. In implementations that need to maintain the bidirectionality of the graph index structure, the construction system can also trigger a reverse connection correction process after generating the target candidate set.

[0120] As a possible implementation, further, based on the sorted candidate set, after truncating and retaining candidate nodes with a preset distance, and generating the target candidate set, the system can calculate the vector distance between each node based on the node vector in the target candidate set, and then dynamically update the occlusion status of subsequent candidate nodes according to the vector distance. The occlusion status is a status marker indicating whether a candidate node is covered by existing neighboring nodes.

[0121] For example, vector distance can characterize the relative proximity of different candidate nodes in the target candidate set in the vector space, and can be calculated using Euclidean distance, cosine distance, or other distance metrics consistent with graph index.

[0122] Optionally, the system can read node vectors from the target candidate set from a vector database or graph index cache, and perform normalization processing under the same computational dimension to ensure consistency in distance comparison. The system can write the vector distance calculation results between each candidate node into a temporary distance matrix for subsequent occlusion state determination.

[0123] For example, the occlusion status can be a marker indicating whether a candidate node has been covered by existing neighboring nodes within its local neighborhood.

[0124] Optionally, after obtaining the target candidate set, the system can mark subsequent candidate nodes whose distance from existing neighbor nodes is less than a threshold range as having an occlusion state, and mark uncovered candidate nodes as having an occlusion state that is not having an occlusion state, based on the vector positions corresponding to existing neighbor nodes.

[0125] Optionally, if the vector distance between nodes in the target candidate set is updated, the construction system can update the occlusion status synchronously. When the coverage relationship between the current candidate node and its existing neighbor nodes changes, the construction system can re-determine whether the subsequent candidate node is still within the occlusion range and adjust its eligibility to participate in the connection calculation accordingly.

[0126] Optionally, the occlusion state can be implemented using a state bit array or a Boolean flag table, where each array element corresponds one-to-one with a candidate node, and the state bit value indicates whether the node continues to participate in neighbor filtering. Regarding the storage format, length, and mapping method of the state array, other models or other data structures can be selected in practical applications; this application does not limit this.

[0127] By calculating the vector distances between nodes within the target candidate set and updating the occlusion status accordingly, the candidate selection process can reflect local neighborhood coverage relationships in a timely manner. This makes the selection of subsequent retained nodes more consistent with the needs of graph structure construction, while reducing the continued participation of invalid candidates. Therefore, while ensuring the quality of neighbor selection, the number of subsequent judgments and redundant vector distance calculations can be reduced, and the execution efficiency of graph index construction can be improved.

[0128] In some embodiments, the system can compute the vector distance between nodes in parallel using hardware parallelization technology. Hardware parallelization technology utilizes the parallel computing capabilities of graphics processing units or field-programmable gate arrays to process multiple computing tasks simultaneously. It can also reduce the data transmission overhead of vector distance computation through memory optimization technology.

[0129] Optionally, the system can divide each node vector in the target candidate set into multiple consecutive data blocks according to the vector dimension, synchronously read and perform vector distance calculations on different data blocks through hardware parallel units, and then summarize the vector distance results between each node by reduction logic.

[0130] Optionally, for the graphics processing unit, the system can split the candidate set into multiple subsets and distribute them to different thread bundles for concurrent processing; for the field-programmable gate array (FPGA), the system can configure the absolute difference, multiply-accumulate, and result accumulation as parallel functional modules, enabling continuous vector distance calculation at the hardware level. To reduce transmission overhead, vector data can be stored in an aligned manner and loaded in batches. The cache stores the target node vector and candidate node vector of the current batch, and only the distance result is written back after the calculation is completed, without repeatedly transmitting the original vector data.

[0131] By using the above method, the vector distance calculation between nodes is parallelized, and on-chip caching reduces redundant memory accesses, enabling the vector distances of multiple candidate nodes to be obtained synchronously. This shortens the time spent in the vector distance calculation stage, reduces latency caused by data transfer, and improves the processing efficiency of candidate sorting and filtering during graph index construction.

[0132] In this application, the existing neighbor set and the newly added candidate set corresponding to the target node are first input into a unified candidate processing path. Then, an ordered candidate sequence is formed based on the distance to the target node, and the ordered sequence is truncated according to a preset distance condition. This constrains both the distance quality and the size of the node set participating in subsequent graph index updates. This processing method addresses the problems of a large number of candidate nodes, high distance computation volume, and redundant sorting and filtering links in large-scale high-dimensional vector retrieval scenarios. By organizing merging, sorting, and truncating into a continuous process according to their dependencies, the target candidate set is limited to a preset range before entering subsequent graph index processing, thereby achieving candidate processing convergence during graph index construction. It should be understood that the above example is merely illustrative and not limiting.

[0133] Figure 2 This is a flowchart illustrating a graph index neighbor pruning method provided in this application. Figure 2 As shown, the method includes:

[0134] (1) Construction of candidate set;

[0135] Optionally, a candidate neighbor set is constructed for the target node, the existing neighbor set of the target node is merged with the newly added candidate node set, and the merged result is deduplicated to eliminate duplicate nodes and self-linked nodes, thereby obtaining the initial candidate set.

[0136] After the candidate set is generated, the candidate set is sorted according to the distance between the candidate nodes and the target node, and only the candidate nodes with smaller distances are retained to participate in the subsequent pruning calculation.

[0137] By pre-constraining the size of the candidate set, the number of nodes participating in the pruning calculation is kept within a limited range, thereby reducing the computational complexity in the subsequent calculations.

[0138] (2) Cache the distance calculation status of neighboring nodes;

[0139] In the traditional graph index pruning process, for each selected neighbor node, it is necessary to rebuild the vector distance calculation state to calculate the distance relationship between the neighbor node and the candidate node, thus generating repeated vector loading and state initialization overhead.

[0140] Optionally, a distance calculation state reuse mechanism can be introduced, whereby the vector representation of the selected neighbor node is loaded and the state is constructed only once, and the state is cached.

[0141] When calculating the distance to multiple candidate nodes in the future, the cached state can be reused directly for calculation, thereby avoiding repeated execution of vector reading and state initialization processes and reducing memory access and I / O overhead.

[0142] (3) Use occlusion factors to record the degree of occlusion between neighbors;

[0143] During the candidate node pruning process, an occlusion factor is introduced to characterize the redundancy of a candidate node relative to its selected neighbors. Specifically, by calculating the distance between the candidate node and the target node, as well as the distance between the candidate node and its selected neighbors, an occlusion evaluation relationship is constructed to determine whether a candidate node is covered by its existing neighbor structure.

[0144] During the pruning process, once a candidate node is added to the final neighbor set, the occlusion status of subsequent candidate nodes is updated in real time based on the influence of that node on subsequent candidate nodes.

[0145] This method transforms the pruning process from the traditional multi-round full traversal update method to an incremental propagation update method in a single round of traversal, thereby reducing the number of repeated scans and lowering computational overhead.

[0146] (4) Traverse the neighbors, determine the degree of occlusion, and choose whether to keep the neighbors;

[0147] After completing the construction and optimization of the candidate set, the candidate set is traversed in order of distance.

[0148] During the traversal, for each candidate node, it is determined whether it meets the inclusion condition based on the current occlusion state. If the condition is met, it is added to the result set, and the occlusion state of subsequent candidate nodes is updated based on this node. If the condition is not met, the node is skipped directly.

[0149] This process continues until the preset limit for the number of neighbors is reached or the candidate set traversal is completed.

[0150] (5) Obtain the list of retained neighbors.

[0151] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0152] Figure 3 A schematic diagram of a graph index construction device provided in this application is shown below. Figure 3 As shown, the graph index construction device 300 provided in this embodiment includes: a recording module 301, an acquisition module 302, and a testing module 303. Wherein,

[0153] The acquisition module 301 is used to acquire the existing neighbor set and the new candidate set of the target node; the existing neighbor set is the set of neighbor nodes that already exist in the graph index of the target node; the new candidate set is the set of neighbor nodes to be screened generated by local search or random sampling.

[0154] The merging module 302 is used to merge the existing neighbor set with the newly added candidate set to generate an initial candidate set;

[0155] The sorting module 303 is used to sort the candidate nodes in the initial candidate set according to the distance between each candidate node and the target node, so as to obtain a sorted candidate set;

[0156] The generation module 304 is used to truncate candidate nodes that retain a preset distance based on the sorted candidate set, and generate a target candidate set.

[0157] Optionally, the merging module 302 is also used to eliminate duplicate nodes in the existing neighbor set and the newly added candidate set through set operations;

[0158] Remove sub-nodes from the existing neighbor set and the newly added candidate set based on the condition; the sub-node is the same as the target node.

[0159] Optionally, the merging module 302 is also used to dynamically adjust the size of the initial candidate set according to the vector distribution density of the target node; the vector distribution density is the density of node distribution in the vector space;

[0160] The size of the initial candidate set is dynamically adjusted based on the real-time system load.

[0161] Optionally, the sorting module 303 is also used to calculate the vector distance between each candidate node in the initial candidate set and the target node; the vector distance is a numerical value of the similarity between the two vectors;

[0162] Based on this vector distance, the candidate nodes in the initial candidate set are sorted in descending order to obtain the sorted candidate set.

[0163] Optionally, the sorting module 303 is also used to calculate the distance based on the topological connectivity between the candidate node and the target node; and to calculate the distance based on the semantic feature labels of the candidate node and the target node.

[0164] Optionally, the sorting module 303 is further configured to sort the candidate nodes in the initial candidate set in descending order based on the vector distance, and before obtaining the sorted candidate set, load the node vectors in the existing neighbor set and initialize the distance calculation state; and cache the distance calculation state initialization.

[0165] Optionally, the generation module 304 is further configured to, based on the sorted candidate set, truncate candidate nodes that retain a preset distance, generate a target candidate set, and then calculate the vector distance between each node based on the node vectors in the target candidate set.

[0166] The occlusion status of subsequent candidate nodes is dynamically updated based on the vector distance. This occlusion status is a status marker indicating whether a candidate node is covered by existing neighbor nodes.

[0167] Optionally, the generation module 304 is also used to calculate the vector distance between nodes in parallel using hardware parallelization technology; the hardware parallelization technology is to use the parallel computing capabilities of a graphics processing unit or a field-programmable gate array to process multiple computing tasks simultaneously.

[0168] Reduce data transfer overhead for vector distance calculations through memory optimization techniques.

[0169] This embodiment provides a graph index construction device that can execute the methods provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0170] Figure 4 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 4 As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 400 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0171] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0172] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0173] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0174] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0175] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0176] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0177] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0178] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0179] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0180] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0181] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in the form of software program modules.

[0182] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0183] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0184] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0185] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0186] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for constructing a graph index, characterized in that, The method includes: Obtain the existing neighbor set and the new candidate set of the target node; the existing neighbor set is the set of neighbor nodes that already exist in the graph index of the target node; the new candidate set is the set of neighbor nodes to be screened generated through local search or random sampling. The existing neighbor set and the newly added candidate set are merged to generate an initial candidate set; The candidate nodes in the initial candidate set are sorted according to their distances to the target node to obtain a sorted candidate set; Based on the sorted candidate set, candidate nodes with a preset distance are truncated to generate a target candidate set.

2. The method according to claim 1, characterized in that, The process of merging the existing neighbor set and the newly added candidate set includes: Duplicate nodes in the existing neighbor set and the newly added candidate set are eliminated through set operations; The self-linked nodes in the existing neighbor set and the newly added candidate set are removed by conditional judgment; the self-linked nodes are the same as the target node.

3. The method according to claim 1, characterized in that, The step of merging the existing neighbor set and the newly added candidate set to generate an initial candidate set includes at least one of the following: The initial candidate set size is dynamically adjusted based on the vector distribution density of the target nodes; the vector distribution density is the density of node distribution in the vector space. The size of the initial candidate set is dynamically adjusted based on the real-time system load.

4. The method according to claim 1, characterized in that, The process of sorting candidate nodes in the initial candidate set according to their distances to the target node yields a sorted candidate set, including: Calculate the vector distance between each candidate node in the initial candidate set and the target node; the vector distance is the numerical value of the similarity between two vectors. The candidate nodes in the initial candidate set are sorted in descending order based on the vector distance to obtain a sorted candidate set.

5. The method according to claim 4, characterized in that, The calculation of the vector distance between each candidate node in the initial candidate set and the target node includes at least one of the following: Calculate the distance based on the topological connectivity between the candidate node and the target node; The distance is calculated based on the semantic feature labels of the candidate node and the target node.

6. The method according to claim 4, characterized in that, Before sorting the candidate nodes in the initial candidate set in descending order based on the vector distance to obtain the sorted candidate set, the following steps are included: Load the node vectors from the existing neighbor set and initialize the distance calculation state; The distance calculation state is initialized and cached.

7. The method according to claim 1, characterized in that, After truncating candidate nodes by retaining a preset distance based on the sorted candidate set, the process of generating the target candidate set includes: Based on the node vectors in the target candidate set, calculate the vector distance between each node; The occlusion status of subsequent candidate nodes is dynamically updated based on the vector distance. The occlusion status is a status marker indicating whether a candidate node is covered by an existing neighbor node.

8. The method according to any one of claims 1-7, characterized in that, The step of calculating the vector distance between nodes based on the node vectors in the target candidate set includes: The vector distance between nodes is calculated in parallel using hardware parallelization technology; the hardware parallelization technology utilizes the parallel computing capabilities of graphics processing units or field-programmable gate arrays to process multiple computing tasks simultaneously. Reduce data transfer overhead for vector distance calculations through memory optimization techniques.

9. A graph index construction apparatus, characterized in that, include: The acquisition module is used to acquire the existing neighbor set and the new candidate set of the target node; the existing neighbor set is the set of neighbor nodes that already exist in the graph index of the target node; the new candidate set is the set of neighbor nodes to be screened generated through local search or random sampling. The merging module is used to merge the existing neighbor set and the newly added candidate set to generate an initial candidate set; The sorting module is used to sort the candidate nodes in the initial candidate set according to the distance between each candidate node and the target node, so as to obtain a sorted candidate set; The generation module is used to truncate candidate nodes by retaining a preset distance based on the sorted candidate set, and generate a target candidate set.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images