Index construction method and device, terminal and computer readable storage medium
By dividing the number of index data groups and determining the target simplification module in the multi-layer index model, the distributed construction of large-scale vector indexes is realized, solving the problem of low construction efficiency of single-institutions and meeting actual needs.
Patent Information
- Application Number
- CN202311344579.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-17
- Publication Date
- 2025-04-18
AI Technical Summary
When there are a lot of vectors, it is very slow to build vector indexing through stand-alone, which is difficult to meet actual needs.
By dividing the number of index data groups based on the vector to be indexed, the target index layer is determined from the multi-layer index model, the target center point is obtained, and the target simplification module is determined based on the center point, and the index interface is called to build the index vector.
It realizes the division of each vector to be indexed into the corresponding simplified module and performs distributed construction, solving the problem of low efficiency of single-institution construction and meeting the needs of large-scale vector index construction.
Smart Images
Figure CN119848038A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to an index construction method, device, terminal and computer-readable storage medium. Background Art
[0002] In recent years, in order to meet the needs of data query and retrieval, data index construction technology has developed rapidly. In data index construction technology, vector-based index construction has become a more common index construction technology due to its high efficiency. However, when there are a large number of vectors, such as more than 10 billion, vector index construction on a single machine is very slow and can no longer meet actual needs. Summary of the invention
[0003] The embodiments of the present application provide an index construction method, device, terminal and computer-readable storage medium, which can solve the technical problem that when there are a large number of vectors, vector index construction by a single machine is very slow and can no longer meet actual needs.
[0004] In a first aspect, an embodiment of the present application provides an index construction method, the method comprising:
[0005] Determine the number of index data groups of the vector to be indexed, and determine a target index layer from a multi-layer index model based on the number of index data groups;
[0006] Obtaining a target center point corresponding to the to-be-indexed vector from the target index layer; wherein each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module;
[0007] Determine a target simplification module corresponding to the vector to be indexed from the simplification modules based on the target center point;
[0008] An index is constructed for the vector to be indexed by calling an index interface based on the target simplification module.
[0009] Optionally, determining the number of index data groups of the vector to be indexed includes:
[0010] The construction amount of the to-be-indexed vector to be indexed is obtained, and the number of index data groups matching the construction amount is determined.
[0011] Optionally, determining a target index layer from a multi-layer index model based on the number of index data groups includes:
[0012] A target index layer whose number of center points matches the number of index data groups is determined from the multi-layer index model.
[0013] Optionally, acquiring the target center point corresponding to the to-be-indexed vector from the target index layer includes:
[0014] Based on the upper index layer in the model grouping configuration file, searching the vector to be indexed layer by layer to obtain a search processing result; wherein the model grouping configuration file includes the target index layer and the upper index layer corresponding to the target index layer in the multi-layer index model;
[0015] Based on the search processing result, a target center point corresponding to the vector to be indexed is determined from the center points of the target index layer.
[0016] Optionally, performing search processing on the to-be-indexed vector layer by layer based on the upper index layer in the model grouping configuration file to obtain the search processing result includes:
[0017] Based on the connection relationship of the center points of each index layer in the upper index layer and the distance information between the center point of each index layer and the vector to be indexed, the vector to be indexed is searched for the cross-layer mapping center point layer by layer until the target cross-layer mapping center point of the tail index layer corresponding to the upper index layer is obtained;
[0018] The determining, based on the search processing result, from the center point of the target index layer and corresponding to the vector to be indexed, comprises:
[0019] A center point corresponding to the target cross-layer mapping center point is determined from the center points of the target index layer, and a target center point corresponding to the vector to be indexed is determined based on the center point corresponding to the target cross-layer mapping center point and the vector to be indexed.
[0020] Optionally, the calling an index interface based on the target simplification module to construct an index for the vector to be indexed includes:
[0021] Based on the target simplification module, an index interface is called to add the vector to be indexed, so as to obtain an index simplification module;
[0022] A target distributed index is obtained based on the index simplification module and the model grouping configuration file.
[0023] Optionally, the method further comprises:
[0024] Determine a lower index layer of the target index layer based on the multi-layer index model;
[0025] A simplification module corresponding to each center point in the target index layer is constructed based on the lower index layer.
[0026] Optionally, the method further comprises:
[0027] Determining the multi-layer index model based on the hierarchical navigable small-world graph;
[0028] determining a target navigable small-world graph from the hierarchical navigable small-world graph based on the number of index data sets,
[0029] The target index level is determined based on the target navigable small-world graph.
[0030] Optionally, determining a target navigable small-world graph from a hierarchical navigable small-world graph based on the number of index data groups comprises:
[0031] A target navigable small-world graph having a number of center points matching the number of index data sets is determined from the hierarchical navigable small-world graph.
[0032] In a second aspect, an embodiment of the present application provides an index construction device, the device comprising:
[0033] A first determination module, adapted to determine the number of index data groups of the vector to be indexed, and determine a target index layer from a multi-layer index model based on the number of index data groups;
[0034] An acquisition module, adapted to acquire a target center point corresponding to the to-be-indexed vector from the target index layer; wherein each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module;
[0035] A second determination module, adapted to determine a target simplification module corresponding to the to-be-indexed vector from the simplification modules based on the target center point;
[0036] The index building module is adapted to build an index for the to-be-indexed vector by calling an index interface based on the target simplification module.
[0037] In a third aspect, an embodiment of the present application provides a terminal, the terminal comprising:
[0038] Processor; and
[0039] A memory arranged to store computer executable instructions, which when executed cause the processor to perform any of the methods described above.
[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by a processor, they implement any of the methods described above.
[0041] The beneficial effects brought about by the technical solutions provided in some embodiments of the embodiments of the present application include at least: first, the number of index data groups is divided based on the vector to be indexed, and the target index layer corresponding to the number of index data groups is found from the multi-layer index model according to the number of index data groups obtained by the division, and the target center point in the target index layer is determined based on the vector to be indexed. Since each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module, the target simplification module corresponding to the vector to be indexed can be determined from the simplification module based on the target center point, and finally, the index interface is called based on the target simplification module to construct an index for the index vector to be indexed. The embodiments of the present application realize the distributed construction of the index vector by dividing each vector to be indexed into the corresponding simplification module by corresponding different center points in the target index layer to different simplification modules, which can solve the technical problem that when there are many vectors, the vector index construction through a single machine is very slow and difficult to meet actual needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 An exemplary system architecture diagram of an index construction method provided in an embodiment of the present application;
[0044] Figure 2 A flowchart of an index construction method provided in an embodiment of the present application;
[0045] Figure 3 A schematic diagram of a process for obtaining a target center point corresponding to a vector to be indexed provided in an embodiment of the present application;
[0046] Figure 4 A schematic diagram of the structure of an HNSW provided in an embodiment of the present application;
[0047] Figure 5 A schematic diagram of a distributed index structure provided in an embodiment of the present application;
[0048] Figure 6 A schematic diagram of a process for constructing an index for an index vector based on a target simplification module calling an index interface according to an embodiment of the present application;
[0049] Figure 7 A schematic diagram of a process for determining a target index layer based on a target navigable small-world graph provided in an embodiment of the present application;
[0050] Figure 8 A schematic diagram of the structure of an index building device provided in an embodiment of the present application;
[0051] Fig. 9 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to make the features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the embodiments of the present application.
[0053] With the continuous development of machine learning technology, its capabilities are becoming increasingly powerful, allowing it to be involved in more and more fields. In many fields, such as vector-related fields, machine learning models can be used to convert things into vectorized forms. As the dimensions of the constructed vectors become higher and higher, and the data scale also shows a trend of increasing, the construction of vector indexes on a single machine is very slow and can no longer meet actual needs. It should be noted that since it is very time-consuming to find similar vectors in tens of millions or even hundreds of billions of vectors, it is necessary to build a certain structure for the vector to speed up the search. This construction process is the vector index construction.
[0054] In the prior art, even if vector index construction is performed based on a distributed cluster, since the data on a single machine is serial, that is, the data between different machines are interrelated, it is impossible to perform parallel construction in the vector index construction, resulting in low construction efficiency of the vector index construction.
[0055] Specifically, in the prior art, the HNSW (Hierarchical Navigable Small World) index is a whole that cannot be split and does not support multi-threaded parallel construction. HNSW is a version that is further optimized based on the NSW (Navigable Small World) algorithm. Its core idea is to introduce a skip list on the basis of the NSW algorithm to implement the idea of layering, thereby further optimizing the retrieval speed of the target vector. The NSW algorithm is a hash-based nearest neighbor search algorithm that achieves fast retrieval by building a multi-layer hash table. However, as the amount of data increases, the retrieval speed of the NSW algorithm may decrease. In order to solve this problem, the HNSW algorithm introduces a skip list to reduce the complexity of retrieval through layering.
[0056] Specifically, the HNSW index organizes data points into a hierarchical structure, each of which contains multiple nodes. These nodes are connected to each other through skip lists, forming a tree-like structure. During the retrieval process, the HNSW index starts from the root node and jumps between the levels through skip lists to quickly find the node where the target vector is located. However, the HNSW index is a whole and cannot be split, and does not support multi-threaded parallel construction.
[0057] Therefore, in the prior art, when there are a large number of vectors, for example, more than 10 billion, constructing a vector index using a single machine is very slow and can no longer meet actual needs.
[0058] In order to solve the above technical problems, an embodiment of the present application provides an index construction method, which first divides the number of index data groups based on the vector to be indexed, finds the target index layer corresponding to the number of index data groups from the multi-layer index model according to the number of index data groups obtained by the division, and determines the target center point in the target index layer based on the vector to be indexed. Since each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module, the target simplification module corresponding to the vector to be indexed can be determined from the simplification module based on the target center point. Finally, the index interface is called based on the target simplification module to construct an index for the index vector to be indexed. In the embodiment of the present application, different center points in the target index layer correspond to different simplification modules, thereby realizing the distributed construction of the vector to be indexed by dividing each vector to be indexed into the corresponding simplification module. This can solve the technical problem that when there are many vectors, vector index construction through a single machine is very slow and difficult to meet actual needs.
[0059] See also Figure 1 , Figure 1 An exemplary system architecture diagram of an index construction method provided in an embodiment of the present application.
[0060] like Figure 1 As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.
[0061] The terminal 101 can interact with the server 103 through the network 102 to receive messages from the server 103 or send messages to the server 103, or the terminal 101 can interact with the server 103 through the network 102 to receive messages or data sent by other users to the server 103. The terminal 101 can be hardware or software. When the terminal 101 is hardware, it can be various terminals, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers and desktop computers. When the terminal 101 is software, it can be installed in the terminals listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.
[0062] The server 103 may be a business server that provides various services. It should be noted that the server 103 may be hardware or software. When the server 103 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or it may be implemented as a single server. When the server 103 is software, it may be implemented as multiple software or software modules (for example, for providing distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.
[0063] In an embodiment of the present application, the server 103 can determine the number of index data groups of the vector to be indexed, and determine the target index layer from the multi-layer index model based on the number of index data groups; then, obtain the target center point corresponding to the vector to be indexed from the target index layer; wherein each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module; then, based on the target center point, determine the target simplification module corresponding to the vector to be indexed from the simplification module; finally, call the index interface based on the target simplification module to build an index for the vector to be indexed.
[0064] It should be understood that Figure 1 The number of terminals, networks and servers in the figure is only for illustration and any number of terminals, networks and servers may be used according to implementation requirements.
[0065] See also Figure 2 , Figure 2 A flowchart of an index building method provided for an embodiment of the present application. The execution subject of the embodiment of the present application can be a server that executes the index building method, or a processor in the server that executes the index building method, or an index building service in the server that executes the index building method. For the convenience of description, the specific execution process of the index building method is introduced below by taking the execution subject being a processor in the server as an example.
[0066] like Figure 2 As shown, the index building method includes:
[0067] S202: Determine the number of index data groups of the vector to be indexed, and determine a target index layer from a multi-layer index model based on the number of index data groups.
[0068] Among them, the vector to be indexed can be a vectorized representation of an image, audio, text, and feature of an object. For example, in image processing, each pixel of an image can be represented as a vector, and we can use these vectors to rotate, scale, translate, and other operations on the image to achieve various transformations of the image. A hierarchical navigable small-world graph can be obtained based on model training. Therefore, the index construction method corresponding to the embodiment of the present application includes but is not limited to constructing corresponding index vectors for images, audio, text, and feature of an object.
[0069] The number of index data groups of the vector to be indexed can be determined based on the construction amount when the vector to be indexed is indexed, so as to improve the index construction efficiency. Of course, the number of index data groups of the vector to be indexed can also be manually determined in advance.
[0070] After determining the number of index data groups of the vector to be indexed, the target index layer can be determined from the multi-layer index model. Here, the multi-layer index model may store the vector to be indexed or may not store the vector to be indexed, and the multi-layer index model may be a multi-level index. Exemplarily, the multi-layer index model may be HNSW, and the target index layer may be the target NSW in the HNSW.
[0071] Exemplarily, determining the number of index data groups of the vector to be indexed in S202 includes:
[0072] The construction amount of the to-be-indexed vector of the to-be-indexed vector is obtained, and the number of index data groups matching the construction amount is determined.
[0073] The number of vectors to be indexed for indexing can be multiple. When the number of vectors to be indexed is large, for example, more than 10 billion, distributed indexing can be performed on the vectors to be indexed. Therefore, the construction amount of the vectors to be indexed for indexing can be obtained first, and the number of index data groups that match the construction amount can be determined based on a mapping table of the number of index data groups and the construction amount of the vectors to be indexed that is pre-set. Each group of index data groups corresponds to a distributed construction node, and multiple groups of index data groups correspond to multiple distributed construction nodes.
[0074] In this embodiment, the number of index data groups for distributed index construction is determined based on the construction amount of the to-be-indexed vectors of the index to be constructed, so that the number of index data groups for distributed construction can be reasonably adjusted based on the construction amount of the to-be-indexed vectors of the index to be constructed, thereby improving the index construction efficiency and saving energy consumption.
[0075] After determining the number of index data groups of the vector to be indexed, illustratively, determining a target index layer from a multi-layer index model based on the number of index data groups in S202 includes:
[0076] Determine a target index layer from the multi-layer index model whose number of center points matches the number of index data groups.
[0077] Based on the number of index data groups of the vector to be indexed, a target index layer whose number of center points matches the number of index data groups is found from the multi-layer index model. Specifically, when the number of index data groups is an interval, the index layers whose number of center points falls within the interval can be screened out from the multi-layer index model, and then the index layers with fewer center points can be selected from the index layers falling within the interval as the target index layers.
[0078] Of course, when the number of index data groups is a specific number, index layers whose center points are greater than or equal to the number of index data groups can be filtered out from the multi-layer index model, and then index layers with fewer center points can be selected from the index layers whose center points are greater than or equal to the number of index data groups as the target index layers.
[0079] S204: Obtain a target center point corresponding to the vector to be indexed from the target index layer; wherein each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module.
[0080] After determining the target index layer, since each center point in the target index layer corresponds to a group of index data and each group of index data corresponds to a simplification module, the target center point in the target index layer can be searched for the index vector according to a preset search path.
[0081] For example, please refer to Figure 3 , Figure 3 A schematic diagram of a process for obtaining a target center point corresponding to a vector to be indexed provided in an embodiment of the present application. Figure 3 As shown, in S204, the target center point corresponding to the vector to be indexed is obtained from the target index layer, including:
[0082] S302: Based on the upper index layer in the model grouping configuration file, search and process the index vector layer by layer to obtain a search and processing result; wherein the model grouping configuration file includes a target index layer and an upper index layer corresponding to the target index layer in the multi-layer index model.
[0083] The model grouping configuration file is constructed based on the target index layer and the upper index layer corresponding to the target index layer in the multi-layer index model. The upper index layer may include one layer, two layers or multiple layers. Specifically, in the multi-layer index model, the upper index layer may be a single layer or two or more layers in succession. It should be noted that the number of layers of each layer in the upper index layer is generally greater than or equal to 1.
[0084] Please refer to Figure 4 , Figure 4 A schematic diagram of the structure of a HNSW provided in an embodiment of the present application. Figure 4 As shown, there are multiple layers of NSW in HNSW. Based on the distributed construction requirements of users, the target NSW can be selected from the multiple layers of NSW in HNSW. Then, the upper layer NSW corresponding to the target NSW is obtained, and the layer number in HNSW that is greater than the layer number of the target NSW is taken as the upper layer NSW.
[0085] For example, the layer number may include 0, 1, and 2. When the layer number of the target NSW is 1, the layer number of the upper NSW is 2. Another example is that the layer number may include 0, 1, 2, and 3. When the layer number of the target NSW is 1, the layer number of the upper NSW is 2 and 3. In this case, the upper NSW includes two layers.
[0086] After determining the upper index layer of the multi-layer index model, a model grouping configuration file is created based on the center point structure in the upper index layer and the target index layer and the center point correspondence between layers. It should be noted that when the target index layer does not have an upper index layer, the model grouping configuration file can be directly obtained based on the target index layer.
[0087] See also Figure 5 , Figure 5 A schematic diagram of the structure of a distributed index provided in an embodiment of the present application. Figure 4 The layer=1 and layer=2 parts of HNSW together constitute the model grouping configuration file, that is, the Part.build part in the figure.
[0088] After determining the model grouping configuration file, based on a method similar to adding a vector to be indexed in a multi-layer index model, the vector to be indexed is searched layer by layer based on the upper index layer to obtain a search result of the vector to be indexed in the upper index layer.
[0089] S304: Based on the search processing result, determine the target center point corresponding to the vector to be indexed from the center points of the target index layer.
[0090] After obtaining the search processing result of the upper index layer for the vector to be indexed, based on the correspondence between layers, the target center point corresponding to the search processing result is determined from the center point of the target index layer based on the search processing result, and the target center point corresponding to the vector to be indexed can be obtained.
[0091] Exemplarily, in S302, based on the upper index layer in the model grouping configuration file, the to-be-indexed vector is searched layer by layer to obtain a search result, including:
[0092] Based on the connection relationship between the center points of each index layer in the upper index layer and the distance information between the center point of each index layer and the vector to be indexed, the cross-layer mapping center point search is performed on the vector to be indexed layer by layer until the target cross-layer mapping center point of the tail index layer corresponding to the upper index layer is obtained.
[0093] When searching for the vector to be indexed layer by layer based on the upper index layer, in the first layer of the upper index layer, that is, the largest layer, first randomly select a center point as the starting center point, and other first-layer center points are connected around the starting center point, and the distance between the first-layer center point connected to the starting center point and the vector to be indexed is calculated. Here, each first-layer center point in the model grouping configuration file has its own corresponding coordinates. After obtaining the distance between the first-layer center point connected to the starting center point and the vector to be indexed, determine the minimum distance, then find the target first-layer center point corresponding to the minimum distance, and then continue to find other first-layer center points connected around the target first-layer center point, and continue to calculate the distance between the first-layer center point connected to the target first-layer center point and the vector to be indexed, until the first-layer target center point closest to the vector to be indexed is found among the first-layer center points.
[0094] After that, the neighboring center point of the first-layer target center point in the next layer is determined based on the neighboring point id, Offset and cum_nneighbor_per_level, and the neighboring center point is used as the starting center point of the next layer to continue the above process until the tail index layer corresponding to the upper index layer is found.
[0095] Among them, the neighboring center point id refers to the neighboring center point of each center point in each layer, which is stored in the array corresponding to the neighboring center point id. The adjacent center points of each center point in all layers are stored adjacently. Offset records the neighboring center points of all layers of each center point, and the starting position in the data corresponding to the neighboring center point id. cum_nneighbor_per_level represents the sum of the number of neighboring center points in the previous layer. Here, the number of neighboring center points of all center points in the same layer is the same. For example, there are 3 layers, and the neighboring center points (layer:num) corresponding to each layer are 0:5, 1:3, and 2:3, then the array is 0:5:8:11, and each layer is the sum of the previous layers, which can be understood as the starting position. After getting the neighboring center point id, the starting position of all neighboring center points corresponding to this id in the neighboring center point id array can be obtained from the offset array. After getting the number of layers, according to cum_nneighbor_per_level, the starting and ending positions of the neighboring center points of the current center point in the current layer can be obtained.
[0096] After searching the tail index layer corresponding to the upper index layer, determine the starting center point in the tail index layer, and calculate the distance between the center point connected to the starting center point and the vector to be indexed based on the starting center point in the tail index layer and the center point connected to the starting center point. After obtaining the distance between the center point connected to the starting center point and the vector to be indexed, determine the minimum distance, then find the initial nearest center point corresponding to the minimum distance, and then continue to find other first-layer center points connected around the initial nearest center point, and continue to calculate the distance between the center point connected to the initial nearest center point and the vector to be indexed, until the target cross-layer mapping center point closest to the vector to be indexed is found among the center points.
[0097] At the same time, in S304, based on the search processing result, the target center point corresponding to the vector to be indexed is determined from the center point of the target index layer, including:
[0098] A center point corresponding to the target cross-layer mapping center point is determined from the center points of the target index layer, and a target center point corresponding to the vector to be indexed is determined based on the center point corresponding to the target cross-layer mapping center point and the vector to be indexed.
[0099] After determining the target cross-layer mapping center point, determine the center point corresponding to the target cross-layer mapping center point from the center point of the target index layer, that is, determine the starting center point from the center point of the target index layer, and other center points are connected around the starting center point, and calculate the distance between the center point connected to the starting center point and the vector to be indexed. After obtaining the distance between the center point connected to the starting center point and the vector to be indexed, determine the minimum distance, then find the center point corresponding to the minimum distance, and then continue to find other center points connected around the center point corresponding to the minimum distance, and continue to calculate the distance between the center point connected to the center point corresponding to the minimum distance and the vector to be indexed, until the target center point closest to the vector to be indexed is found among the center points.
[0100] S206: Determine a target simplification module corresponding to the vector to be indexed from the simplification modules based on the target center point.
[0101] Since each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module, the target simplification module corresponding to the vector to be indexed can be determined from the simplification modules based on the target center point.
[0102] S208: Calling the index interface based on the target simplification module to build an index for the to-be-indexed vector.
[0103] In an embodiment of the present application, the index construction can rely on the distributed processing framework and the mapping module therein, and each target simplification module constructs a part of the index. This can give full play to the advantages of the cluster. As long as there are sufficient resources, an index of any size can be constructed quickly and efficiently.
[0104] For example, Figure 6 As shown, Figure 6 A schematic diagram of a process of constructing an index for an index vector based on a target simplification module calling an index interface in an embodiment of the present application. In S208, constructing an index for an index vector based on a target simplification module calling an index interface includes:
[0105] S602: Based on the target simplification module, the index interface is called to add the vector to be indexed, and the index simplification module is obtained.
[0106] Among them, the lower index layer of the target index layer can be determined based on the multi-layer index model, and then a simplification module corresponding to each center point in the target index layer is constructed based on the lower index layer.
[0107] For example, Figure 4 As shown, when the lower index layer of the target index layer is the lower NSW corresponding to the target NSW, where the HNSW has multiple layers of NSW, the target NSW can be selected from the multiple layers of NSW in the HNSW based on the user's distributed construction requirements. After that, the lower NSW corresponding to the target NSW is obtained, and the layer number in the HNSW is less than the layer number of the target NSW as the lower NSW corresponding to the target NSW.
[0108] For example, the layer number may include 0, 1 and 2. When the layer number of the target NSW is 1, the layer number of the lower NSW is 0. For another example, the layer number may include 0, 1, 2 and 3. When the layer number of the target NSW is 2, the layer number of the lower NSW is 0, 1 and 3. In this case, the lower NSW includes two layers.
[0109] Of course, in some other embodiments, the HNSW can be created based on a multi-layer NSW including a lower layer NSW. Figure 5 As shown, for example, the multi-layer NSW including the lower layer NSW includes but is not limited to a multi-layer index layer.
[0110] Specifically, the target simplification module can be created based on the lower index layer of the target index layer in the multi-layer index model and each center point in the target index layer. That is, each center point in the target index layer has a corresponding simplification module, and each simplification module is used to construct the index data corresponding to the center point. Each center point in the target index layer is located in the same layer in the model grouping configuration file, so that the index data constructed by each center point can not interfere with each other. Of course, in other embodiments, each center point can be located in a different layer in the model grouping configuration file, and at this time, the construction of the index data by each center point can overlap with each other to achieve independent distributed construction of the index data.
[0111] Since the target simplification module is created based on the lower index layer of the target index layer in the multi-layer index model and each center point in the target index layer, after determining the target center point corresponding to the vector to be indexed from each center point in the target index layer of the model grouping configuration file, the target simplification module corresponding to the target center point can be determined from multiple simplification modules based on the target center point, and then the vector to be indexed is added to the target simplification module to obtain the index simplification module.
[0112] Different target simplification modules can be set in different servers. Since the index data constructed by each target center point does not interfere with each other, different servers can construct index data based on the target simplification modules in a distributed manner at the same time.
[0113] In the embodiment provided by the present application, the to-be-indexed vectors are divided according to the model grouping configuration file, and the to-be-indexed vectors to be indexed are merged into the target simplification module corresponding to the target center point to which the to-be-indexed vectors are close, and then the index interface of the target simplification module is called to build the index. After the index is built, the indexes corresponding to the multiple target simplification modules are placed under one path to form a complete index. Figure 5 Each part can independently build an index and provide retrieval. In the prior art, since the vector index is a whole, it can only be rebuilt if it needs to be expanded. Now the vector index on each machine contains multiple parts. When expanding, some parts can be quickly migrated to other machines.
[0114] Specifically, in S602, when the target simplification module calls the index interface to add the vector to be indexed, the first index layer of the target simplification module is determined based on the index interface, and the end center point corresponding to the vector to be indexed is determined from the first center point of the first index layer.
[0115] For example, based on the index interface, the adjacent center point corresponding to the target mapping center point in the first-level index layer of the target simplification module is determined as the starting center point in the first-level index layer, and then the ending center point in the first-level index layer is determined based on the starting center point and the vector to be indexed.
[0116] Afterwards, the starting center point in the secondary index layer corresponding to the first index layer is determined based on the ending center point, the ending center point in the secondary index layer is determined based on the starting center point in the secondary index layer and the vector to be indexed, and the search is continued based on the ending center point in the secondary index layer until the starting center point in the tail index layer in the target simplification module is determined.
[0117] Among them, after obtaining the end center point in the first-level index layer, the adjacent center point corresponding to the end center point in the second-level index layer is determined as the starting center point in the second-level index layer based on the end center point in the first-level index layer, and then the end center point in the second-level index layer is determined based on the starting center point in the second-level index layer and the vector to be indexed, and similar steps are repeated in the next layer until the starting center point in the tail-level index layer in the target simplification module is finally determined.
[0118] Then, based on the starting center point and the vector to be indexed in the tail index layer, the center point to be added to the vector to be indexed in the tail index layer is determined, and the vector to be indexed is added to the center point to be added to the vector to be indexed.
[0119] Among them, after determining the starting center point in the tail index layer, based on the center point connection structure in the tail index layer, calculate the center point closest to the vector to be indexed, and use the center point closest to the vector to be indexed in the tail index layer as the center point to be added. Finally, the vector to be indexed is added to the center point to be added to the vector to be indexed.
[0120] S604: Obtain a target distributed index based on the index simplification module and the model grouping configuration file.
[0121] The number of index simplification modules can be multiple, and different index simplification modules can be distributed in different servers to achieve distributed distribution. Based on the model grouping configuration file, the vectors to be indexed are divided into multiple index simplification modules. Different index simplification modules are used to build index data of different target center points. Therefore, they will not affect each other during the distributed construction process, thereby solving the technical problem that when there are many vectors, the vector index construction through a single machine is very slow and difficult to meet actual needs.
[0122] like Figure 7 As shown, Figure 7 A schematic diagram of a process for determining a target index layer based on a target navigable small-world graph provided in an embodiment of the present application. In an embodiment provided in the present application, the method includes:
[0123] S702: Determine a multi-layer index model based on the hierarchical navigable small-world graph.
[0124] Among them, the multi-layer index model can be obtained based on the trained HNSW.
[0125] S704: Determine a target navigable small-world graph from the layered navigable small-world graph based on the number of index data groups, and determine a target index layer based on the target navigable small-world graph.
[0126] Among them, a target navigable small-world graph whose number of center points matches the number of index data groups can be determined from the hierarchical navigable small-world graph.
[0127] In an embodiment provided in the present application, an index search may be performed based on the obtained target distributed index. For example, first, a vector to be queried is generated in response to a search request.
[0128] The server receives the search request, extracts features in the search request, and generates a query vector based on the features in the search request. The search request includes but is not limited to an image search request, a text search request, and an audio and video search request.
[0129] Afterwards, based on the model grouping configuration file in the target distributed index, a first target retrieval center point corresponding to the query vector is determined.
[0130] Based on the query vector, the target center point of the model grouping configuration file in the target distributed index is searched, and the center point of the query vector closest to the target center point is searched from the target center points as the first target retrieval center point.
[0131] Then, based on multiple simplification modules in the target distributed index, a target simplification module corresponding to the first target retrieval center point is determined, and the vector to be queried is queried based on the target simplification module.
[0132] After determining the first target retrieval center point, find the target simplification module corresponding to the target retrieval center point from multiple simplification modules in the target distributed index, and then search for the vector to be queried from the target simplification module. In one embodiment provided in the present application, querying the vector to be queried based on the target simplification module includes: querying layer by layer in the target simplification module, determining the second target retrieval center point corresponding to the vector to be queried in the tail index layer of the target simplification module, and querying the vector to be queried based on the second target retrieval center point. Figure 5 As shown in the figure, according to the part.build file, if there is a Q center point query, the nearest first target retrieval center point p is located, and the index corresponding to the p center point is in part2. Because the p center point is at layer=1, the Q center point at layer 0 can be located in part2, and then the search is performed based on the Q center point.
[0133] The embodiment of the present application is based on the target distributed index, determines the first target retrieval center point corresponding to the vector to be queried from the target center point of the model grouping configuration file, and then finds the target simplification module corresponding to the first target retrieval center point, thereby avoiding traversing all simplification modules to search for the vector to be queried, and improving the query efficiency of the vector to be queried.
[0134] An embodiment of the present application further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.
[0135] See also Figure 8 , Figure 8 A schematic diagram of the structure of an index building device provided in an embodiment of the present application.
[0136] The index building device 800 includes:
[0137] A first determination module 810, adapted to determine the number of index data groups of the vector to be indexed, and determine a target index layer from a multi-layer index model based on the number of index data groups;
[0138] The acquisition module 820 is adapted to acquire the target center point corresponding to the to-be-indexed vector from the target index layer; wherein each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module;
[0139] A second determination module 830, adapted to determine a target simplification module corresponding to the vector to be indexed from the simplification modules based on the target center point;
[0140] The index building module 840 is adapted to build an index for the to-be-indexed vector based on calling the index interface by the target simplification module.
[0141] Optionally, the first determining module 810 is adapted to obtain a construction amount of the to-be-indexed vectors to be indexed, and determine the number of index data groups matching the construction amount.
[0142] Optionally, the first determination module 810 is adapted to determine, from the multi-layer index model, a target index layer whose number of center points matches the number of index data groups.
[0143] Optionally, the acquisition module 820 includes:
[0144] A layer-by-layer search unit, adapted to perform a search process on the to-be-indexed vector layer by layer based on the upper index layer in the model grouping configuration file to obtain a search process result; wherein the model grouping configuration file includes a target index layer and an upper index layer corresponding to the target index layer in the multi-layer index model;
[0145] The determination unit is adapted to determine, based on the search processing result, a target center point corresponding to the vector to be indexed from the center points of the target index layer.
[0146] Optionally, the layer-by-layer search unit is adapted to perform cross-layer mapping center point search for the to-be-indexed vector layer by layer based on the center point connection relationship of each index layer in the upper index layer and the distance information between the center point of each index layer and the to-be-indexed vector, until the target cross-layer mapping center point of the tail index layer corresponding to the upper index layer is obtained;
[0147] The determination unit is adapted to determine a center point corresponding to a target cross-layer mapping center point from among center points of a target index layer, and determine a target center point corresponding to the vector to be indexed based on the center point corresponding to the target cross-layer mapping center point and the vector to be indexed.
[0148] Optionally, the index building module 840 includes:
[0149] An adding unit, adapted to add a vector to be indexed based on a target simplification module by calling an index interface to obtain an index simplification module;
[0150] The target distributed index determination unit is adapted to obtain the target distributed index based on the index simplification module and the model grouping configuration file.
[0151] Optionally, the index building device 800 includes:
[0152] A third determination module, adapted to determine a lower index layer of the target index layer based on the multi-layer index model;
[0153] The construction module is adapted to construct a simplification module corresponding to each center point in the target index layer based on the lower index layer.
[0154] Optionally, the index building device 800 includes:
[0155] a fourth determination module adapted to determine a multi-layer index model based on a hierarchical navigable small-world graph;
[0156] The target index layer determination module is adapted to determine a target navigable small-world graph from the hierarchical navigable small-world graph based on the number of index data groups, and to determine a target index layer based on the target navigable small-world graph.
[0157] Optionally, the target index layer determination module is adapted to determine, from the hierarchical navigable small-world graph, a target navigable small-world graph whose number of center points matches the number of index data groups.
[0158] See also Fig. 9 , Fig. 9 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. Fig. 9 As shown, the terminal 900 may include: at least one processor 901 , at least one network interface 904 , a user interface 903 , a memory 905 , and at least one communication bus 902 .
[0159] The communication bus 902 is used to realize the connection and communication between these components.
[0160] The user interface 903 may include a display screen (Display) and a camera (Camera), and the optional user interface 903 may also include a standard wired interface and a wireless interface.
[0161] The network interface 904 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0162] Among them, the processor 901 may include one or more processing cores. The processor 901 uses various interfaces and lines to connect various parts within the entire terminal 900, and executes various functions and processes data of the terminal 900 by running or executing instructions, programs, code sets or instruction sets stored in the memory 905, and calling data stored in the memory 905. Optionally, the processor 901 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 901 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 901, and it can be implemented separately through a chip.
[0163] Among them, the memory 905 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 905 includes a non-transitory computer-readable storage medium. The memory 905 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 905 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 905 may optionally be at least one storage device located away from the aforementioned processor 901. As Fig. 9 As shown, the memory 905 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an index building program.
[0164] exist Fig. 9 In the terminal 900 shown, the user interface 903 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 901 can be used to call the index building program stored in the memory 905 and specifically perform the following operations:
[0165] Determine the number of index data groups of the vector to be indexed, and determine a target index layer from a multi-layer index model based on the number of index data groups;
[0166] Obtaining a target center point corresponding to the vector to be indexed from the target index layer; wherein each center point in the target index layer is used to correspond to a set of index data, and each set of index data corresponds to a simplification module;
[0167] Determine a target simplification module corresponding to the vector to be indexed from the simplification modules based on the target center point;
[0168] Based on the target simplification module, the index interface is called to build an index for the index vector.
[0169] Optionally, when the processor 901 determines the number of index data groups of the vector to be indexed, it specifically performs:
[0170] The construction amount of the to-be-indexed vector of the to-be-indexed vector is obtained, and the number of index data groups matching the construction amount is determined.
[0171] Optionally, when the processor 901 determines the target index layer from the multi-layer index model based on the number of index data groups, it specifically performs:
[0172] Determine a target index layer from the multi-layer index model whose number of center points matches the number of index data groups.
[0173] Optionally, when the processor 901 obtains the target center point corresponding to the vector to be indexed from the target index layer, it specifically executes:
[0174] Based on the upper index layer in the model grouping configuration file, search and process the to-be-indexed vector layer by layer to obtain a search and process result; wherein the model grouping configuration file includes a target index layer and an upper index layer corresponding to the target index layer in the multi-layer index model;
[0175] Based on the search processing result, a target center point corresponding to the vector to be indexed is determined from the center points of the target index layer.
[0176] Optionally, the processor 901 performs search processing on the to-be-indexed vector layer by layer based on the upper index layer in the model grouping configuration file, and when the search processing result is obtained, specifically performs:
[0177] Based on the connection relationship of the center points of each index layer in the upper index layer, and the distance information between the center point of each index layer and the vector to be indexed, the cross-layer mapping center point search is performed on the vector to be indexed layer by layer until the target cross-layer mapping center point of the tail index layer corresponding to the upper index layer is obtained;
[0178] When the processor 901 determines the target center point corresponding to the vector to be indexed from the center point of the target index layer based on the search processing result, it specifically executes:
[0179] A center point corresponding to the target cross-layer mapping center point is determined from the center points of the target index layer, and a target center point corresponding to the vector to be indexed is determined based on the center point corresponding to the target cross-layer mapping center point and the vector to be indexed.
[0180] Optionally, when the processor 901 executes the target simplification module to call the index interface to build an index for the index vector, the processor 901 specifically executes:
[0181] Based on the target simplification module, the index interface is called to add the vector to be indexed, and the index simplification module is obtained;
[0182] The target distributed index is obtained based on the index simplification module and the model grouping configuration file.
[0183] Optionally, the processor 901 further executes:
[0184] Determine a lower index layer of a target index layer based on a multi-layer index model;
[0185] A simplification module corresponding to each center point in the target index layer is constructed based on the lower index layer.
[0186] Optionally, the processor 901 further executes:
[0187] Determine a multi-layer index model based on a hierarchical navigable small-world graph;
[0188] A target navigable small-world graph is determined from the hierarchical navigable small-world graph based on the number of index data groups, and a target index layer is determined based on the target navigable small-world graph.
[0189] Optionally, when the processor 901 determines the target navigable small-world graph from the hierarchical navigable small-world graph based on the number of index data groups, it specifically executes:
[0190] A target navigable small-world graph is determined from the hierarchical navigable small-world graph, the number of center points of which matches the number of index data groups.
[0191] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0192] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0193] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.
[0194] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.
[0195] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.
[0196] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0197] The above is a description of an index construction method, device, terminal and computer-readable storage medium provided in an embodiment of the present application. For technical personnel in this field, based on the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.
Claims
1. A method for constructing an index, wherein: The method comprises: Determine the number of index data groups of the vector to be indexed, and determine a target index layer from a multi-layer index model based on the number of index data groups; Obtaining a target center point corresponding to the to-be-indexed vector from the target index layer; wherein each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module; Determine a target simplification module corresponding to the vector to be indexed from the simplification modules based on the target center point; An index is constructed for the vector to be indexed by calling an index interface based on the target simplification module.
2. The method according to claim 1, wherein: The step of determining the number of index data groups of the vector to be indexed includes: The construction amount of the to-be-indexed vector to be indexed is obtained, and the number of index data groups matching the construction amount is determined.
3. The method according to claim 1, wherein: The determining a target index layer from a multi-layer index model based on the number of index data groups includes: A target index layer whose number of center points matches the number of index data groups is determined from the multi-layer index model.
4. The method according to claim 1, wherein: The acquiring the target center point corresponding to the to-be-indexed vector from the target index layer includes: Based on the upper index layer in the model grouping configuration file, searching the vector to be indexed layer by layer to obtain a search processing result; wherein the model grouping configuration file includes the target index layer and the upper index layer corresponding to the target index layer in the multi-layer index model; Based on the search processing result, a target center point corresponding to the vector to be indexed is determined from the center points of the target index layer.
5. The method according to claim 4, wherein: The step of searching the vectors to be indexed layer by layer based on the upper index layer in the model grouping configuration file to obtain a search result includes: Based on the connection relationship of the center points of each index layer in the upper index layer and the distance information between the center point of each index layer and the vector to be indexed, the vector to be indexed is searched for the cross-layer mapping center point layer by layer until the target cross-layer mapping center point of the tail index layer corresponding to the upper index layer is obtained; The determining, based on the search processing result, from the center point of the target index layer and corresponding to the vector to be indexed, comprises: A center point corresponding to the target cross-layer mapping center point is determined from the center points of the target index layer, and a target center point corresponding to the vector to be indexed is determined based on the center point corresponding to the target cross-layer mapping center point and the vector to be indexed.
6. The method according to claim 4, wherein: The step of calling an index interface based on the target simplification module to construct an index for the vector to be indexed includes: Based on the target simplification module, an index interface is called to add the vector to be indexed, so as to obtain an index simplification module; A target distributed index is obtained based on the index simplification module and the model grouping configuration file.
7. The method according to claim 1, wherein: The method further comprises: Determine a lower index layer of the target index layer based on the multi-layer index model; A simplification module corresponding to each center point in the target index layer is constructed based on the lower index layer.
8. An index building device, wherein: The device comprises: A first determination module, adapted to determine the number of index data groups of the vector to be indexed, and determine a target index layer from a multi-layer index model based on the number of index data groups; An acquisition module, adapted to acquire a target center point corresponding to the to-be-indexed vector from the target index layer; wherein each center point in the target index layer is used to correspond to a group of index data, and each group of index data corresponds to a simplification module; A second determination module, adapted to determine a target simplification module corresponding to the to-be-indexed vector from the simplification modules based on the target center point; The index building module is adapted to build an index for the to-be-indexed vector by calling an index interface based on the target simplification module.
9. A terminal, wherein: The terminal includes: Processor; and A memory arranged to store computer executable instructions which, when executed, cause the processor to perform a method according to any one of claims 1 to 7.
10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, which, when executed by a processor, implement the method of any one of claims 1 to 7.