A method and apparatus for retrieving a vector of a proximity graph combining a hard disk and a memory

Abstract

The present application relates to a kind of hard disk and the near neighbor graph vector retrieval method and device of memory, first according to the distance of each feature vector in data set and constructs near neighbor graph, then it is saved in hard disk and is released in memory according to 4K alignment mode optimization its storage and resource, then according to the structure information of near neighbor graph saved in hard disk, calculate sample set, again construct the near neighbor graph of sample set object as sample near neighbor graph.Using query vector on sample near neighbor graph executes nearest neighbor search to obtain the preliminary more similar query target set, finally with the more similar query target set obtained as the initial solution of near neighbor graph in hard disk, executes nearest neighbor search to obtain the most similar query target.The present application reduces the memory occupation in large-scale data search process under the condition of guaranteeing accuracy, and improves the search efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of vector retrieval, and particularly relates to a method and device for retrieving a vector of a near neighbor graph combining a hard disk and a memory. BACKGROUND

[0002] With the development of artificial intelligence, vector retrieval technology is widely used in face recognition, file search, speech recognition and other applications. Among them, the vector retrieval technology based on the near neighbor graph has become a research hotspot in the current academic and industrial circles due to its excellent search navigation ability.

[0003] However, due to the fact that the current vector retrieval technology based on the near neighbor graph relies too much on the memory, with the sharp increase in the data size, the high memory consumption and the high computing cost pose great challenges to the wide application of the near neighbor graph vector retrieval. In order to solve this problem, a main solution method currently adopted by the industry is to combine the hard disk and the memory for vector retrieval, so as to greatly reduce the memory occupation during vector retrieval by storing the near neighbor graph mainly occupying the memory resources in the hard disk. In order to reduce the adverse effects of the hard disk access on the retrieval delay, the mainstream method is to construct a product quantization index independent of the near neighbor graph in the memory for the vector data, so as to serve as an addressing aid to accelerate the index positioning and reading. However, this method has performance problems caused by the low quantization calculation accuracy and the still relatively high number of hard disk access times. In view of this problem, the present application proposes a method and device for retrieving a vector of a near neighbor graph combining a hard disk and a memory. SUMMARY

[0004] Based on the above-mentioned shortcomings and deficiencies in the prior art, the present application provides a method for retrieving a vector of a near neighbor graph combining a hard disk and a memory.

[0005] S1, obtaining a vector data set S, and constructing a near neighbor graph G according to the similarity distances between the vectors in the vector data set S

[0006] S2, adjusting the storage structure of the near neighbor graph G according to the 4K alignment mode according to the near neighbor relationship and saving it in the hard disk, to obtain a near neighbor graph G1;

[0007] S3, performing a sampling calculation on the near neighbor graph G1 to obtain a sampling data set M of the vector data set S;

[0008] S4, constructing a sampling near neighbor graph N according to the similarity distances between the vectors in the sampling data set M;

[0009] S5, obtaining a feature vector of a query object as a query vector V, and querying the C objects closest to the query vector V on the sampling near neighbor graph N in S4;

[0010] S6, taking the C objects obtained in S5 as a candidate object set, further querying in the proximity graph G1 in S2 to obtain K reference objects closest to the query vector V, and outputting the K reference objects as the result of vector retrieval.

[0011] As a preferred solution, the 4K alignment method mentioned in step S2 is specifically:

[0012] S2.1, calculating the disk space SZ required for a single node in the proximity graph G stored in the adjacency list, in the form of (unit: byte):

[0013] SZ=T1D+4P+4

[0014] Wherein, T1 is the space occupied by a unit data type, such as 4 bytes for single-precision floating point type, 8 bytes for double-precision floating point type, etc., D is the dimension of the vector data, and P is the upper limit of the edges of each node in the proximity graph; wherein each node in the proximity graph G includes: a vector corresponding to the current node only, a neighbor node number of the current node, and a number of the current node itself.

[0015] S2.2, traversing all unselected nodes x in the proximity graph G, calculating the node size n of node x that can be stored in a 4K block, and the specific calculation method is:

[0016]

[0017] If n>0, x and the neighbor point of x closest to x and not selected are stored in the same 4K block, and these points are marked as selected; if n=0, node x is stored in a continuous 4K block alone, and x is marked as selected;

[0018] S2.3, when all nodes in the proximity graph G are marked as selected, release the memory occupied by the proximity graph G.

[0019] As a preferred solution, step S3 is specifically:

[0020] S3.1, setting the vector set M', initializing the vector set M' as the vector data set S, i.e. putting all vectors in the vector data set S into the vector set M', and setting the memory limit E;

[0021] S3.2, randomly selecting a vector v1 in the vector set M' that has not been selected, and marking the vector v1 as selected;

[0022] S3.3, in the proximity graph G1 obtained in S2, reading the 4K block where the node x1 corresponding to the vector v1 is located, obtaining the neighbor set of the node x1, and removing the vectors corresponding to the nodes in the neighbor set of the node x1 that are included in the vector set M'.

[0023] S3.4, repeat steps S3.2 and S3.3 until all vectors in the vector set M' are marked as selected;

[0024] S3.5, calculate the expected size S of the neighbor graph generated by the vector set M' g = |M'| x SZ, where |M'| is the number of vectors contained in the vector set M', and SZ is the disk space required by the single node as described in step S2.1;

[0025] S3.6, compare the expected size S of the neighbor graph with the size of the set memory limit E, if the expected size S of the neighbor graph is greater than the set memory limit E, mark all vectors in the vector set M' as unselected, repeat steps S3.2-S3.5, otherwise, end the sampling, and return the current vector set M' as the sampled data set M. g g S3.6, compare the expected size S of the neighbor graph with the size of the set memory limit E, if the expected size S of the neighbor graph is greater than the set memory limit E, mark all vectors in the vector set M' as unselected, repeat steps S3.2-S3.5, otherwise, end the sampling, and return the current vector set M' as the sampled data set M.

[0026] As a further preferred solution, the specific way of the sampling process in step S3 can be adjusted according to the characteristics of the search space, i.e. the clustering of the vector data set S; if the vector data set S is unknown distribution or biased uniform distribution, the sampling can be performed in the way as described in steps S3.1 to S3.6; if the vector data set S is found to have clustering clusters C i satisfying the condition the specific way of the sampling process includes the following steps:

[0027] S'3.1, set a candidate vector set M c and a vector set M0, initialize the candidate vector set M c as the vector data set S, initialize the vector set M0 as an empty set, and set the memory limit E;

[0028] S'3.2, randomly select a vector v2 in the candidate vector set M c which has not been selected, and mark the vector v2 as selected;

[0029] S'3.3, in the neighbor graph G1 obtained in S2, read the 4K block in which the node x2 corresponding to the vector v2 is located, obtain the neighbor set of the node x2, and remove the vectors corresponding to the nodes contained in the neighbor set of the node x2 from the candidate vector set M c ;

[0030] S'3.4, repeat steps S'3.2 and S'3.3 until all elements in the candidate vector set M c are marked as selected;

[0031] S'3.5, calculate the expected size S of the neighbor graph generated by the candidate vector set M c ;​g =|M c |×SZ, where |M c | for candidate vector set M c The number of vectors included, SZ is the disk space required for a single node as described in step S2.1;

[0032] S'3.6, compare the expected size S of the nearest neighbor graph. g The size of the nearest neighbor graph is determined by the set memory limit E. g If the value exceeds the memory limit E, then the candidate vector set M will be... c Mark all vectors as unselected and repeat the steps.

[0033] S'3.2~S'3.5, otherwise, update the vector set M0=M0∩M c Update memory limit E = ES g ;

[0034] S'3.7, Initialize the candidate vector set M c For all conditions that are met The intersection of clusters Repeat steps S'3.2 to S'3.6 to end the sampling and return the current vector set M0 as the sampling dataset M.

[0035] Where |C i |For clustering C i The number of vectors within the vector dataset S;

[0036] In addition, if the user knows the nearest neighbor range of the vector to be queried in advance... Or a user on a subset C of a vector dataset S s For more precise query requirements, the sampling method of S'3 can also be used to view C. r and C s Sample the vector set that meets the conditions.

[0037] The present invention also provides a nearest neighbor graph vector retrieval device combining hard disk and memory, comprising:

[0038] A device for constructing a nearest neighbor graph G based on the similarity distance between vectors in a vector dataset S;

[0039] The device that adjusts the storage structure of the nearest neighbor graph G according to the nearest neighbor relationship in 4K alignment and saves it to the hard disk to obtain the nearest neighbor graph G1;

[0040] A device for performing a sampling process on the nearest neighbor graph G1 to obtain a sampled dataset M of the vector dataset S;

[0041] An apparatus for constructing a sample neighbor graph N according to the similarity distance between each vector in the sample data set M;

[0042] An apparatus for obtaining a feature vector of a query object as a query vector V, and querying C objects closest to the query vector V on the sample neighbor graph N;

[0043] An apparatus for further querying K reference objects closest to the query vector V in the neighbor graph G1 using the C objects as a candidate object set, and outputting the K reference objects as the result of vector retrieval.

[0044] Compared with the prior art, the method has the advantages that: the method of the present application stores the aggregated nodes in the hard disk according to the node neighbor information of the neighbor graph in step S2, thereby reducing the memory occupation and improving the efficiency of reading the neighbor graph in the hard disk; steps S3 and S4 are used to establish a small-scale neighbor graph in the memory as a navigation according to the distribution characteristics of the data set, thereby speeding up the positioning speed of the query result; steps S5 and S6 are used to divide the search process into a rough navigation stage and an accurate search stage, thereby reducing the routing times in the hard disk during the search and shortening the delay under the condition of ensuring the accuracy of the search result. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of a neighbor graph vector retrieval method combining the hard disk and the memory according to an embodiment of the present application;

[0046] Figure 2 is an example flowchart of a neighbor graph vector retrieval method combining the hard disk and the memory according to an embodiment of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application, the specific implementation manner of the present application will be described below with reference to the drawings. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0048] Embodiment 1

[0049] The neighbor graph vector retrieval method and apparatus combining the hard disk and the memory provided in this embodiment have the method flowchart as shown in Figure 1 The present embodiment takes the search of the patent data with unknown type distribution based on the neighbor graph as an example, and the further example flowchart is shown in Figure 2 .

[0050] Before step S1, first generate the corresponding feature vector set for the acquired patent data set. The generation of the feature vector adopts a pre-trained model, and for text data such as patent specifications, an existing BERT model can be used for training to extract features and convert them into a patent feature vector set S;

[0051] Then step S1 is performed, and a patent neighbor graph G1 is constructed according to the similarity distance of each vector in the patent feature vector set S. The calculation method of the similarity distance can be the Lagrange distance between vectors: Where a and b are any D-dimensional vectors, a i is the i-th component of the a vector, b i is the i-th component of the b vector, and the dis function returns the similarity distance between a and b.

[0052] Further, the specific process of constructing the patent neighbor graph G can refer to the construction method of the HNSW graph with a layer number of 1:

[0053] S1.1, select a patent vector e i from the patent feature vector set S a , select efc objects with the smallest distance or greater distance than all neighbor patent vectors currently connected to it as the vector set S a , and set the edge upper limit P;

[0054] S1.2, select a patent vector with the smallest distance from the vector set S i and connect it to e i ;

[0055] S1.3, select patent vectors from the vector set S a in order from close to far from the patent vector e j and connect them to the patent vector e i , until the number of edges of the object vector e i reaches the edge upper limit P or all patent vectors in the vector set S a have been tried;

[0056] S1.4, determine whether there are still patent vectors that have not been selected in step S1.1, if yes, return to step S1.1, if no, complete the generation of the patent neighbor graph G(VT, EG), where VT is the node set and EG is the edge set.

[0057] Then step S2 is performed, and the neighbor structure of the patent neighbor graph G is adjusted in the 4K alignment manner, and saved in the hard disk to obtain the neighbor graph G1;

[0058] Further, the specific process of the 4K alignment manner is:

[0059] S2.1, calculate the disk space SZ required for a single patent node in the proximity graph stored in the adjacency list, in the following manner (in bytes):

[0060] SZ = T1D + 4P + 4

[0061] wherein T1 is the space occupied by a unit data type, such as 4 bytes for single-precision floating point type, 8 bytes for double-precision floating point type, etc., D is the dimension of the vector data, and P is the upper limit of the edges of each node in the proximity graph; wherein each node in the proximity graph G includes: a vector corresponding to the current node only, the neighbor node number of the current node, and the number of the current node itself.

[0062] S2.2, randomly select a point x in the proximity graph G that has not been selected, and calculate the node size n that can be stored in a 4K block about the node x, in the following manner:

[0063]

[0064] If n > 0, store x and the neighbor node of x closest to x and not selected in the same 4K block, and mark these nodes as selected; if n = 0, store node x alone in a continuous 4K block, and mark x as selected.

[0065] S2.3, when all the points in the proximity graph are marked as selected, release the memory occupied by the patent proximity graph G.

[0066] Then, step S3 is performed, which is a sampling calculation on the patent proximity graph G1 described in S2, to obtain a sampling patent feature vector set M of the patent feature vector set S.

[0067] Further, the generation process of the sampling patent feature vector set M is as follows:

[0068] S3.1, set the vector set M', initialize the vector set M' as the patent feature vector set S, and set the memory limit E;

[0069] S3.2, randomly select a vector v1 in the vector set M' that has not been selected, and mark the vector v1 as selected;

[0070] S3.3, read the 4K block in which the node x1 corresponding to the vector v1 is located in the patent proximity graph G1 obtained in S2, obtain the neighbor set of the node x1, and remove the vectors in the vector set M' corresponding to the nodes contained in the neighbor set of x1.

[0071] S3.4, repeat steps S3.2 and S3.3 until all the vectors in the vector set M' are marked as selected.

[0072] ​S3.5, calculate the expected size S of the neighbor graph generated by the vector set M' g = |M'| x SZ, where |M'| is the number of vectors contained in the vector set M', and SZ is the space occupied by the single node in step S2.1;

[0073] S3.6, compare the expected size S of the neighbor graph g with the size of the set memory limit E, if the expected size S of the neighbor graph is greater than the size of the set memory limit E, then mark all vectors in the vector set M' as unselected, repeat steps S3.2-S3.5, otherwise, end the sampling, and return the current vector set M' as the sampled patent feature vector set M. g s S3.6, compare the expected size S of the neighbor graph g with the size of the set memory limit E, if the expected size S of the neighbor graph is greater than the size of the set memory limit E, then mark all vectors in the vector set M' as unselected, repeat steps S3.2-S3.5, otherwise, end the sampling, and return the current vector set M' as the sampled patent feature vector set M.

[0074] Then, step S4 is performed to construct a sampled patent neighbor graph N according to the similarity distance between each vector in the sampled patent feature vector set M according to the method in S1; this method is the same as the method in step S1 above, with the difference being that the parameters are different, S1 is for constructing a graph for the feature vector set S, and S4 is for constructing a graph for the sampled feature vector set.

[0075] At this point, the neighbor graph generation phase is complete, and the neighbor graphs G1 and N can be used for querying. Before step S5 is performed, a query object is generated from the content of the patent to be queried as a query vector V, i.e., a feature vector generated by a pre-trained model for the text data of the patent specification to be queried.

[0076] When the feature vector of the patent data to be queried is generated, step S5 is performed to query the C closest patent feature vector objects on the sampled patent neighbor graph N using the query vector V.

[0077] Finally, step S6 is performed to use the C patent feature vector objects obtained in the previous step as a candidate object set to further query the K closest patent feature vector objects in the neighbor graph G1, and the patent objects corresponding to the obtained patent feature vectors are the top K most similar patent objects of the patent object to be queried.

[0078] Further, the query algorithm uses a commonly used greedy algorithm, which is as follows:

[0079] S5.1, randomly select a node of the sampled patent neighbor graph N as a query starting node e s , and add it to the candidate node set B, and set the maximum capacity of the candidate node set B as ef;

[0080] S5.2, select the unvisited node x2 with the smallest distance from V from B, and obtain the neighbor node set Nei(x) = (y1, y2, …, y l ) of x2, where y lthe lth neighbor node of x2, l is the number of neighbors of x2; mark x2 as a visited node;

[0081] S5.3, add each neighbor node in Nei(x) to B, and only keep the first ef nodes in B with the minimum distance to V;

[0082] S5.4, determine whether there is an unvisited node in B, if yes, return to step S5.2, if not, end the query and return the first C nodes in B with the minimum distance to V as the query result.

[0083] In addition, it needs to be additionally explained that the sampling method mentioned in step S3 of the method of embodiment 1 can be flexibly adjusted according to the scene requirements. The sampling method adopted in embodiment 1 is considering that the distribution of the feature vector set S of the patent is unknown, otherwise, the highly clustered data clusters can be emphatically sampled according to the clustering situation of the data, for example, embodiment 2.

[0084] Embodiment 2

[0085] The method and device for retrieving a near neighbor graph vector combining a hard disk and a memory provided in the embodiment are shown in the method flowchart as shown in the figure, and in the embodiment, the patent data containing most software-related patent content is searched based on a near neighbor graph, and the flowchart is further illustrated as shown in the figure. Figure 1 Figure 2

[0086] Before step S1, first generate the corresponding feature vector set for the obtained patent data set. The generation of the feature vector adopts a pre-trained model, and for patent specification and other text data, the existing BERT model can be used for training to extract features and convert into a patent feature vector set S;

[0087] Then, step S1 is performed, and a patent near neighbor graph G1 is constructed according to the similarity distance of each vector in the patent feature vector set S. The calculation method of the similarity distance can be the Lagrange distance between vectors: where a and b are any D-dimensional vectors, a i is the ith component of the a vector, b i is the ith component of the b vector, and the dis function returns the similarity distance of a and b;

[0088] Further, the specific process of constructing the patent near neighbor graph G can refer to the construction method of the HNSW graph with the number of layers being 1:

[0089] S1.1, select a patent vector e i from the patent feature vector set S, select efc objects with the minimum distance or greater than the distance of all neighbor patent vectors currently connected to it as the vector set S​​a Set the upper limit P for connecting edges;

[0090] S1.2, from the vector set S a Select one that is related to the patent vector e i Connect it to the patent vector with the smallest distance;

[0091] S1.3, with respect to the distance patent vector e i The order from nearest to farthest starts from the vector set S. a Selecting patent vector e j With patent vector e i Connect the edges until the object vector e is reached. i The number of edges reaches the upper limit P or the vector set S is exhausted. a All patent vectors in the database;

[0092] S1.4 Determine if there are still patent vectors that were not selected in step S1.1. If yes, return to step S1.1. Otherwise, complete the generation of the patent nearest neighbor graph G(VT,EG), where VT is the node set and EG is the edge set.

[0093] Then proceed to step S2, adjust its storage in 4K alignment according to the nearest neighbor structure of the patent nearest neighbor graph G and save it in the hard disk to obtain the nearest neighbor graph G1.

[0094] Furthermore, the specific process for 4K alignment is as follows:

[0095] S2.1 Calculate the disk space SZ required for a single patent node in the nearest neighbor graph stored according to the adjacency list, calculated in bytes:

[0096] SZ = T1D + 4P + 4

[0097] Where T1 is the space occupied by the unit data type, such as 4 bytes for single-precision floating-point type and 8 bytes for double-precision floating-point type, D is the dimension of the vector data, and P is the upper limit of the edges of each node in the nearest neighbor graph; where each node in the nearest neighbor graph G includes: a vector uniquely corresponding to the current node, the neighbor node numbers of the current node, and the number of the current node itself.

[0098] S2.2, randomly select an unselected point x in the nearest neighbor graph G, and calculate the node size n that can be stored in a 4K block for node x. The specific calculation method is as follows:

[0099]

[0100] If n > 0, then x and its nearest unselected neighboring points are stored in the same 4K block, and these points are marked as selected; if n = 0, then node x is stored separately in... Within a consecutive 4K block, mark x as selected;

[0101] Since most of the patent content in the known obtained patent dataset is software-related, performing KMeans clustering on the generated patent feature vector set S reveals the existence of a software cluster C0 that satisfies the condition. Therefore, step S3 is adjusted to perform a sampling calculation on the patent nearest neighbor graph G described in S2 to obtain the sampled patent feature vector set M of the patent feature vector set S;

[0102] Furthermore, the process of generating the sampled patent feature vector set M is as follows:

[0103] S'3.1, Set the candidate vector set M c Given a vector set M0, initialize the candidate vector set M. c Given a vector dataset S, initialize the vector set M0 as an empty set and set the memory limit E;

[0104] S'3.2, Randomly select candidate vector set M c If vector v2 has not been selected, mark vector v2 as selected.

[0105] S'3.3, in the nearest neighbor graph G1 obtained in S2, read the 4K block where the node x2 corresponding to vector v2 is located, obtain the neighbor set of node x2, and remove the candidate vector set M. c The vectors corresponding to the nodes contained in the neighbor set of node x2;

[0106] S'3.4, Repeat steps S'3.2 and S'3.3 until the candidate vector set M is reached. c All vectors in the array are marked as selected;

[0107] S'3.5, Calculate according to candidate vector set M c The expected size S of the generated nearest neighbor graph g =|M c |×SZ, where |M c |For the candidate vector set M c The number of vectors included, SZ is the disk space required for a single node as described in step S2.1;

[0108] S'3.6, compare the expected size S of the nearest neighbor graph. g The size of the nearest neighbor graph is determined by the set memory limit E. g If the value exceeds the memory limit E, then the candidate vector set M will be... c Mark all vectors as unselected and repeat the steps.

[0109] S'3.2~S'3.5, otherwise, update the vector set M0=M0∩M c Update memory limit E = ESg ;

[0110] S'3.7, initialize candidate vector set M c For the clustering cluster set C0, repeat steps S'3.2-S'3.6, end the sampling, and return the current vector set M0 as the sampling patent feature vector set M.

[0111] Where |C i | is the number of elements in the clustering cluster set C i | is the number of elements in the vector data set S;

[0112] Then, step S4 is performed to construct a sampling patent neighbor graph N according to the similarity distance between each vector in the sampling patent feature vector set M according to the method in S1;

[0113] At this point, the neighbor graph generation stage ends, and the neighbor graphs G1 and N can be used for querying. Before step S5 is performed, the content of the patent to be queried is generated as a query object as a query vector V, i.e., a feature vector about the text data of the patent specification generated by the pre-training model.

[0114] When the feature vector of the patent data to be queried is generated, step S5 is performed to query the closest C patent feature vector objects on the sampling patent neighbor graph N using the query vector V;

[0115] Finally, step S6 is performed to use the C patent feature vector objects obtained in the previous step as a candidate object set to further query the closest K patent feature vector objects in the neighbor graph G1, and the patent objects corresponding to the obtained patent feature vectors are the top K most similar patent objects of the patent object to be queried.

[0116] Further, the query algorithm uses a commonly used greedy algorithm, specifically:

[0117] S5.1, randomly select a node of the sampling patent neighbor graph N as a query starting node e s , add it to the candidate node set B, and set the maximum capacity of the candidate node set B as ef;

[0118] S5.2, select the unvisited node x2 with the smallest distance from V from B, and obtain the neighbor node set Nei(x) of x2=(y1, y2,…, y l ), where y l is the lth neighbor node of x2, and l is the number of neighbors of x2; mark x2 as a visited node;

[0119] S5.3, add each neighbor node in Nei(x) to B, and only keep the first ef nodes in B with the smallest distance from V;

[0120] S5.4, judging whether there is a node not visited in B, if yes, returning to step S5.2, if not, ending the query and returning the first C nodes with the minimum distance to V in B as the query result.

[0121] A hard disk and memory combined neighbor graph vector retrieval device for implementing the above-mentioned hard disk and memory combined neighbor graph vector retrieval method, comprising the following devices:

[0122] A neighbor graph construction device for constructing a neighbor graph G according to the similarity distance between each vector in a vector data set S;

[0123] An adjusting device for adjusting the storage structure of the neighbor graph G according to the neighbor relationship in a 4K alignment manner and saving in a hard disk, obtaining an adjusted neighbor graph G1;

[0124] A sampling device for performing sampling processing on the neighbor graph G1, obtaining a sampling data set M of the vector data set S;

[0125] A neighbor graph construction device for constructing a sampling neighbor graph N according to the similarity distance between each vector in the sampling data set M;

[0126] A query device for obtaining a feature vector of a query object as a query vector V and querying C objects closest to the query vector V on the sampling neighbor graph N;

[0127] An output device for taking the C objects as a candidate object set, further querying K reference objects closest to the query vector V in the neighbor graph G1 and outputting the K reference objects as the result of vector retrieval.

[0128] The device executes the above-mentioned hard disk and memory combined neighbor graph vector retrieval method and can achieve the following beneficial effects: the neighbor graph navigation effect can be adjusted according to the set memory limit, a large amount of memory can be saved during searching, the searching delay can be shortened and the accuracy of the result can be maintained, the initial candidate object set can be expanded by calculating the query result of the sampling neighbor graph in the memory in advance, and the storage of the gathered nodes in the hard disk according to the node neighbor information can improve the hard disk reading efficiency and further speed up the searching speed.

[0129] It should be noted that the above only describes the preferred embodiments and principles of the present application, and for those skilled in the art, the specific implementation manner can be changed according to the idea provided by the present application, and these changes should be regarded as the protection scope of the present application.

Claims

1. A method for vector retrieval of a proximity graph combining a hard disk and a memory, characterized in that, The method comprises the following steps: S1, obtaining a vector dataset S, and constructing a neighbor graph G according to the similarity distance between each vector in the vector dataset S; S2, adjusting the storage structure of the neighbor graph G according to the neighbor relationship in a 4K alignment manner and saving in a hard disk to obtain a neighbor graph G1; S3, performing sampling processing on the neighbor graph G1 to obtain a sampling dataset M of the vector dataset S; S4, constructing a sampling neighbor graph N according to the similarity distance between each vector in the sampling dataset M; S5, obtaining a feature vector of a query object as a query vector V, and querying C objects closest to the query vector V on the sampling neighbor graph N in S4; S6, taking the C objects obtained in S5 as a candidate object set, further querying K reference objects closest to the query vector V in the neighbor graph G1 in S2, and outputting the K reference objects as the result of vector retrieval; The specific method for adjusting the neighbor graph G in the step S2 comprises the following steps: S2.1, calculating the disk space SZ required for a single node in the neighbor graph G stored in an adjacency list, and the calculation method is as follows: SZ=T1D+4P+4 Wherein, T1 is the space occupied by a unit data type, D is the dimension of the vector data, and P is the upper limit of the edge of each node of the neighbor graph; each node in the neighbor graph G comprises a vector corresponding to the current node, a neighbor node number of the current node, and a number of the current node itself; S2.2, traversing all unselected nodes x in the neighbor graph G, and calculating the node size n of the node x that can be stored in a 4K block, and the specific calculation method is as follows: If n > 0, store x and the nearest neighbor of x that is not already selected in the same 4K block and mark these points as selected; if n = 0, store x alone in a contiguous 4K block and mark x as selected. If n > 0, store x and the nearest neighbor of x that is not already selected in the same 4K block and mark these points as selected; if n = 0, store x alone in a contiguous 4K block and mark x as selected. S2.3, when all points in the neighbor graph G are marked as selected, releasing the memory occupied by the neighbor graph G.

2. The neighbor graph vector retrieval method combined with a hard disk and a memory according to claim 1, wherein The sampling processing in the step S3 comprises the following steps: S3.1, setting a vector set M', initializing the vector set M' as the vector dataset S, i.e. putting all vectors in the vector dataset S into the vector set M', and setting a memory limit E; S3.2, randomly selecting a vector v1 in the vector set M' that has not been selected, and marking the vector v1 as selected; S3.3, in the neighbor graph G1 obtained in S2, reading the 4K block where the node x1 corresponding to the vector v1 is located, obtaining the neighbor set of the node x1, and removing the vectors corresponding to the nodes in the neighbor set of the node x1 from the vector set M'; S3.4, repeating steps S3.2 and S3.3 until all vectors in the vector set M' are marked as selected; S3.5, compute the expected size S of the neighbor graph generated by the vector set M' g = |M'| x SZ, where |M'| is the number of vectors contained in the vector set M' and SZ is the disk space required by the individual node as described in step S2.

1. S3.6, compare the expected size S of the proximity graph with the size of the set memory limit E g If the expected size S of the proximity graph is greater than the size of the set memory limit E, then mark all vectors in the vector set M' as unselected, repeat steps S3.2-S3.5, otherwise, end the sampling and return the current vector set M' as the sampled data set M. g If the expected size S of the proximity graph is greater than the size of the set memory limit E, then mark all vectors in the vector set M' as unselected, repeat steps S3.2-S3.5, otherwise, end the sampling and return the current vector set M' as the sampled data set M.

3. The neighbor graph vector retrieval method combined with a hard disk and a memory according to claim 1, wherein According to the aggregation of the vector dataset S in the step S1, the specific sampling method of the neighbor graph G1 in the step S3 is adjusted.

4. The neighbor graph vector retrieval method combined with a hard disk and a memory according to claim 3, comprising: When the aggregation of the vector dataset S in the step S1 is that the vector dataset S is clustered by KMeans, there are cluster sets C i satisfy the condition where |C i is the number of elements in the cluster set C i |S| is the number of elements in the vector dataset S, the collection processing in the step S3 specifically includes the following steps: S'3.1, set the candidate vector set M c and vector set M0, initialize the candidate vector set M c for vector dataset S, initialize the vector set M0 as an empty set, set the memory limit E; S'3.2, randomly select a set of candidate vectors M c vectors v2 that have not been selected in the current iteration, mark the vectors v2 as selected; S'3.3, in the 4K block where the node x2 corresponding to the read vector v2 is located in the proximity graph G1 obtained in S2, obtaining the neighbor set of the node x2, removing the candidate vector set M from the vectors corresponding to the nodes contained in the neighbor set of the node x2 c of the node x2 S'3.4, repeat steps S'3.2 and S'3.3 until the vectors in the candidate vector set M c are all marked as selected; S'3.5, Calculate according to candidate vector set M c The expected size S of the generated nearest neighbor graph g =|M c |×SZ, where |M c | for candidate vector set M c The number of vectors included, SZ is the disk space required for a single node as described in step S2.1; S'3.6, compare the expected size S of the nearest neighbor graph. g The size of the nearest neighbor graph is determined by the set memory limit E. g If the value exceeds the memory limit E, then the candidate vector set M will be... c Mark all vectors in the set as unselected and repeat steps S'3.2 to S'3.5; otherwise, update the vector set M0 = M0∩M c Update memory limit E = ES g ; S'3.7, Initialize the candidate vector set M c For all conditions that are met The intersection of clusters Repeat steps S'3.2 to S'3.6 to end the sampling and return the current vector set M0 as the sampled dataset M.

5. A vector retrieval apparatus for a proximity graph of a hard disk and a memory, characterized by comprising: ​ A neighbor graph construction device for constructing a neighbor graph G according to the similarity distance between each vector in a vector dataset S; An adjusting device for adjusting the storage structure of the neighbor graph G according to the neighbor relationship in a 4K alignment manner and saving in a hard disk to obtain a neighbor graph G1; A sampling device for performing sampling processing on the neighbor graph G1 to obtain a sampling dataset M of the vector dataset S; A neighbor graph construction device for constructing a sampling neighbor graph N according to the similarity distance between each vector in the sampling dataset M; A querying device for obtaining a feature vector of a query object as a query vector V and querying C objects closest to the query vector V on the sampling neighbor graph N; An output device for taking the C objects as a candidate object set, further querying K reference objects closest to the query vector V in the neighbor graph G1, and outputting the K reference objects as a result of vector retrieval; A specific method for adjusting the neighbor graph G in the 4K alignment manner, including the following steps: S2.1, calculating the disk space SZ required for a single node in the neighbor graph G stored in an adjacency list, in a calculation manner as follows: SZ=T1D+4P+4 wherein, T1 is the space occupied by a unit data type, D is the dimension of the vector data, and P is the upper limit of the edge of each node of the neighbor graph; each node in the neighbor graph G includes a vector uniquely corresponding to the current node, a neighbor node number of the current node, and a number of the current node itself; S2.2, traversing all unselected nodes x in the neighbor graph G, and calculating the node size n of the node x that can be stored in a 4K block in a specific calculation manner as follows: If n > 0, store x and the nearest neighbor of x that is not already selected in the same 4K block and mark these points as selected; if n = 0, store x alone in a contiguous 4K block and mark x as selected. If n > 0, store x and the nearest neighbor of x that is not already selected in the same 4K block and mark these points as selected; if n = 0, store x alone in a contiguous 4K block and mark x as selected. S2.3, when all points in the neighbor graph G are marked as selected, releasing the memory occupied by the neighbor graph G.

Images