A method, system, device and medium for constructing a k-nearest neighbor graph

Abstract

The application discloses a K-neighbor graph construction method, system, device and medium, wherein original data is quantized to reduce the bit required for representing each data; in the collection process of the nodes of the reverse neighbor list rnn_new and rnn_old, unnecessary node storage is avoided; new flag bits flag(new') and old') are used to represent the nodes originally stored in the existing method NN-Descent, so that no additional data structure is needed to store the nodes in the K-neighbor graph construction process, and the memory consumption is reduced; the reverse neighbor list rnn_new and rnn_old are processed in blocks, the cache miss frequency of the CPU is reduced, and the data access efficiency is improved; the system, device and medium are used for implementing the method, and the K-neighbor graph construction efficiency is improved on the basis of ensuring the high recall rate of the K-neighbor graph, and the memory required for constructing the K-neighbor graph is reduced.

Description

Technical Field

[0001] This invention relates to the technical field of data mining and machine learning, and in particular to a method, system, device, and medium for constructing a K-nearest neighbor graph. Background Technology

[0002] In recent years, K-nearest neighbor (KNN) graph construction has been widely applied in data mining and machine learning. The construction process of a KNN graph can be summarized as follows: Given a dataset D, find the K nearest neighbors (K) for each node u in D. The similarity between data points needs to be measured by a metric, such as Euclidean distance. KNN graphs are frequently used in techniques such as KNN querying, dimensionality reduction, clustering, and outlier detection. KNN querying is widely used in various information retrieval and recommendation systems; for example, image-based product search on shopping platforms is an application of this technology. Among current KNN querying techniques, DPG (Diversified Proximity Graph) and NSG (Navigating Spreading-out Graph) are the latest graph indexing query techniques, and their construction relies on building a KNN graph. With societal development, the amount of data generated in production and daily life is rapidly increasing, and this data often has high dimensionality. To observe the data more intuitively, it is usually necessary to map data from high-dimensional space to low-dimensional space, thereby visualizing the data. This requires dimensionality reduction. However, simply observing data is usually insufficient; we often need to discover the information contained within it. Clustering is a crucial technique for discovering patterns and exploring relationships within data. For example, businesses can segment customers based on their purchasing habits and develop different marketing strategies for different groups. Similar to the two K-nearest neighbor query methods mentioned above, K-nearest neighbor graphs are also a prerequisite for some dimensionality reduction and clustering methods. Furthermore, K-nearest neighbor graphs are also applied in distance-based anomaly detection. Therefore, K-nearest neighbor graphs have wide applications in data mining and machine learning, and improving their performance has significant application value and meaning.

[0003] The existing method for constructing an approximate K-neighbor graph is NN-Descent (Dong W, Moses C, Li K. Efficient k-nearest neighbor graph construction for generic similarity measures [C] / / Proceedings of the 20th international conference on World wide web. 2011: 577-586.), which mainly updates the initial randomly generated K-neighbor graph through join and update operations to obtain a K-neighbor graph with high recall rate. However, the NN-Descent method requires a large amount of memory and is slow in construction. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a K-neighbor graph construction method, system, device and medium, which reduces the bit required to represent each data by quantizing the original data; avoids saving unnecessary nodes during the collection of nodes in the reverse neighbor list rnn_new and rnn_old; compared with the NN-Descent method in the prior art, the new flag bit flag (new' and old') is used to represent the nodes that are originally stored in the neighbor list nn_new and nn_old, so that no additional data structure is needed to save nodes during the K-neighbor graph construction process, reducing the consumption of memory; by performing block processing on the reverse neighbor list rnn_new and rnn_old, the number of cache misses in the CPU is reduced, the data access efficiency is improved, and on the basis of ensuring the high recall rate of the K-neighbor graph, the construction efficiency of the K-neighbor graph is improved, and the memory required for constructing the K-neighbor graph is reduced.

[0005] In order to achieve the above-mentioned purposes, the technical solutions adopted by the present application are as follows:

[0006] A K-neighbor graph construction method, comprising the following steps:

[0007] Step 1: reading the data set D from the user-specified hard disk storage location to the memory, and quantizing the data set D;

[0008] Step 2: initializing the neighbor candidate set pool and the reverse neighbor list rnn_new of each node u in the quantized data set D obtained in step 1;

[0009] Step 3, join operation: distance calculation is performed on the neighbor candidate set pool of each node u, different nodes of the reverse neighbor list rnn_new and rnn_old, and the neighbor candidate set pool of the corresponding node of the neighbor candidate set pool of each node u is updated according to the calculation result;

[0010] Step 4: whether the recall rate set by the user or the number of times of performing the join operation is satisfied is judged, if yes, the iteration is ended, and the first K nodes u are selected from the neighbor candidate set pool of each node u in the last iteration K , to obtain the K-neighbor graph of the data set D; if not, the next step is entered; the parameter K is used to control the out-degree of the K-neighbor graph, that is, the number of neighbors to be searched by each node;

[0011] Step 5, update operation: the flag of each node in the neighbor candidate set pool of each node u and the reverse neighbor list rnn_new and rnn_old are updated, and after traversing each node u in the data set D, the step 3 is returned.

[0012] The specific process of the step 1 is as follows:

[0013] The data set D is read from the hard disk storage location specified by the user to the memory, and the data set D contains n nodes u in a d-dimensional space; in the data set D, each node u is represented by a d-dimensional vector; according to the number n of nodes u, each vector in the data set D is read n times, and the maximum value max and the minimum value min in the nxd values of the entire data set D are recorded; each vector in the data set D is read again n times, and each value is quantized to obtain a new value, and the new value constitutes a quantized data set D which is saved in the memory; the quantization formula is as follows:

[0014]

[0015] Wherein, v represents the original value, nv represents the new value; max represents the maximum value of the nxd values in the data set D, and min represents the minimum value of the nxd values in the data set D.

[0016] The specific process of the step 2 is as follows:

[0017] S nodes u are randomly selected from the n nodes u of the quantized data set D s , the selected nodes u s are different from the current node u i , the distance between the current node u i and the S nodes u s is calculated, and the neighbor candidate set pool of the corresponding node of the neighbor candidate set pool of each node u is updated according to the distance between the current node u iFor S nodes u in order from nearest to farthest s Sort the nodes to obtain an initialized neighbor candidate set pool. The flag bit of each node u in the initialized neighbor candidate set pool is in the state of new.

[0018] Randomly select 2S nodes u from the n nodes u of the quantized dataset D. s , 2S nodes u s Fill directly into the current node u i From the reverse neighbor list rnn_new, we obtain the initialized reverse neighbor list rnn_new.

[0019] The specific process of step 3 is as follows:

[0020] If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of new or old, then skip the current node u. i ;

[0021] If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of 'new', then the node u will be... i And arranged at node u i The flag following it indicates whether the state is 'new' or 'old', or whether the node u in the reverse neighbor lists rnn_new and rnn_old is new. j Perform distance calculations separately; based on node u i and node u j The distance between them, using node u i Go update node u j The neighbor candidate set pool, while using node u j Go update node u i The neighbor candidate set pool; the updated neighbor candidate set pool of node u contains nodes u. i The flag flag is reset to old;

[0022] If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of 'old', then the node u will be... i And arranged at node u i The flag following the node is in the state of 'new' and the node u in the reverse neighbor list rnn_new. j Perform distance calculations separately; based on node u i and node u j The distance between them, using node u iupdating the neighbor candidate set pool of the node u j using the node u j updating the neighbor candidate set pool of the node u i resetting the state of the flag of the node u i in the updated neighbor candidate set pool of the node u to old;

[0023] Meanwhile, distance calculation is performed on different nodes in the reverse neighbor list rnn_new and rnn_old of the node u respectively, and before the distance calculation, the reverse neighbor list rnn_new and rnn_old of the node u is subjected to block processing to obtain sub-blocks of the reverse neighbor list rnn_new and rnn_old of the node u, and the number of the sub-blocks is and respectively, wherein num(rnn_new) and num(rnn_old) represent the number of nodes in the reverse neighbor list rnn_new and rnn_old of the node u respectively, and NIOB represents the number of nodes stored in each sub-block, and the sub-blocks of the reverse neighbor list rnn_new are marked as N1, N2,..., N i , and the sub-blocks of the reverse neighbor list rnn_old are marked as O1, O2,..., O j ; the calculation formula of the number of nodes NIOB stored in each sub-block is as follows:

[0024] NIOB = L2CacheSize / 2 / (dim*2+sizeof(Neighbor)) / 2

[0025] wherein L2CacheSize represents the size of L2 cache in CPU, dim represents the dimension of node vector, and sizeof(Neighbor) represents the size of space occupied by each Neighbor in the neighbor candidate set pool, and the Neighbor includes the id of the current node, the distance from the current node u to a node different from the current node u, and the flag of the current node u;

[0026] two sub-blocks in all the sub-blocks of the reverse neighbor list rnn_new and rnn_old are traversed, and distance calculation is performed on all the nodes u i and u j in the two sub-blocks; according to the distance between the nodes u i and u j , the node u i is used to update the neighbor candidate set pool of the node u j , and the node u j is used to update the neighbor candidate set pool of the node u iThe candidate set of neighbors, pool;

[0027] The two sub-blocks include the following cases:

[0028] (1) The same sub-blocks (N) in the reverse neighbor list rnn_new i With N i If the same sub-block (N) i With N i If the number of nodes in a given array is m, then the number of distance calculations is . Second-rate;

[0029] (2) Different sub-blocks of the reverse neighbor list rnn_new (N i1 With N i2 If different sub-blocks (N) i1 With N i2 If the number of nodes in the distance calculation is m and n respectively, then the distance calculation will be performed m × n times.

[0030] (3) Sub-blocks (N) of the reverse neighbor list rnn_new i ) and the sub-blocks of rnn_old (O j ), if sub-block (N i ) and sub-blocks (O) j If the number of nodes in the distance calculation is m and n respectively, then the distance calculation will be performed m×n times.

[0031] In step 3, according to node u i and node u j The distance between them, using node u j Go update node u i The operation process of the neighbor candidate set pool and the use of node u i Go update node u j The operation process of the neighbor candidate set pool is the same; the use of node u j Go update node u i The specific process of pooling the candidate neighbor set is as follows:

[0032] First, determine node u. j Is it already at node u? i In the neighbor candidate set pool, if node u j Already at node u i If the neighbor candidate set pool is used, then skip node u. j If node u j Not in node u i In the neighbor candidate set pool, determine u i Whether the number of nodes in the neighbor candidate set pool is less than L, where L is the maximum capacity of the neighbor candidate set pool;

[0033] If node u i If the number of nodes in the neighbor candidate set pool is less than L, then node u will be added to the list. j Insert into u i The appropriate position of node u in the neighbor candidate set pool. i The nodes in the neighbor candidate set pool are ranked according to their distance from node u. i Arranged from nearest to farthest, node u j The flag is in a new state;

[0034] If node u i If the number of nodes in the neighbor candidate set pool is not less than L, then the comparison node u i and node u j Distance between nodes u i and node u i The relationship between the distances between the last nodes in the neighbor candidate set pool;

[0035] If node u i and node u i In the neighbor candidate set pool, the distance between the last node is greater than that between nodes u. i and node u j If the distance between them is zero, then delete node u. i The node that is last in the candidate set of neighbors in the pool, and then node u. j Insert into node u i The appropriate position of node u in the neighbor candidate set pool. i The nodes in the neighbor candidate set pool are ranked according to their distance from node u. i Arranged from nearest to farthest, node u j The flag is in a new state;

[0036] If node u i and node u i The distance between the last nodes in the neighbor candidate set pool is no greater than that between nodes u. i and node u j If the distance between them is such that node u is skipped directly. j .

[0037] The specific process of step 5 is as follows:

[0038] For each node u in the dataset D in step 2, clear the reverse neighbor list rnn_new and rnn_old of node u, and update the variable M according to the parameter S in step 2 and the flag status of each node u in the neighbor candidate set pool in step 3, so that at most S nodes in the first M nodes of the neighbor candidate set pool in step 3 are marked as new.

[0039] For the first M nodes u of the neighbor candidate set pool in step 3 M If the current node u M If the flag bit state is "new", then reset the flag bit state to "new'" and store the corresponding node u from dataset D into the corresponding node u. M In the reverse neighbor list rnn_new; if the current node u M If the flag state is 'old', then reset the flag state to 'old' and store node u from dataset D into the corresponding node u. M The reverse neighbor list rnn_old; until the first M nodes u in the neighbor candidate set pool of node u have been traversed. M After traversing each node u in the dataset D, return to step 3.

[0040] The specific process for storing data in step 5 is as follows:

[0041] Determine the current node u M The relationship between the number of nodes in the reverse neighbor list rnn_new or rnn_old and the parameter R is determined. If the number of nodes is less than R, then node u is directly inserted into the current node u. M The last node in the reverse neighbor list rnn_new or rnn_old; if the number of nodes is not less than R, a random integer rand is obtained, and from the random integer rand being 0 to the node u being stored in the current node. M The random integer rand is an integer between R and R's reverse neighbor list, where R is the number of times R is generated. If the random integer rand is less than R, then node u is used to replace the current node u. M The reverse neighbor list rnn_new or rnn_old contains the rand-th node; the parameter R is used to represent the maximum number of nodes that the reverse neighbor lists rnn_new and rnn_old can contain.

[0042] The present invention also provides a system for constructing a K-nearest neighbor graph, comprising:

[0043] Quantization module: Used to quantize the dataset D read from the user-specified hard drive storage location into memory;

[0044] Initialization module: Used to initialize the neighbor candidate set pool and reverse neighbor list rnn_new for each node u in the quantized dataset D;

[0045] a join module: used for calculating the distance between different nodes of the neighbor candidate set pool of each node u, the reverse neighbor list rnn_new and rnn_old, and updating the neighbor candidate set pool of the corresponding node of the neighbor candidate set pool of each node u, the reverse neighbor list rnn_new and rnn_old of node u according to the calculation result;

[0046] a judging module: used for judging whether the recall rate set by a user or the number of times of performing the join operation is satisfied, if yes, the iteration is ended, and the first K nodes u are selected from the neighbor candidate set pool of each node u of the last iteration K , to obtain the K-neighbor graph of the data set D; if no, the update module is entered;

[0047] the update module: used for updating the flag of each node in the neighbor candidate set pool of each node u, and the reverse neighbor list rnn_new and rnn_old, and returning to the join module after traversing each node u in the data set D.

[0048] The application further provides a K-neighbor graph construction device, which comprises:

[0049] a memory: used for storing the computer program of the K-neighbor graph construction method, and used for storing the computer program which is readable by a computer;

[0050] a processor: used for executing the computer program to realize the K-neighbor graph construction method.

[0051] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can realize the K-neighbor graph construction method when the computer program is executed by a processor.

[0052] Compared with the prior art, the application has the beneficial effects that:

[0053] 1. The application reduces the bit required for representing each data by quantizing the original data, reduces the memory space required for saving the data set to half of that required by using the NN-Descent method in the prior art, and enables more data points to be saved in the cache due to the less bit required for saving each data, thereby reducing the number of cache misses in the CPU when the CPU loads data, and improving the K-neighbor graph construction efficiency;

[0054] 2. The application judges the number of nodes in the reverse neighbor list rnn_new and rnn_old in real time in the collection process of the nodes in the reverse neighbor list rnn_new and rnn_old, when the number of nodes is less than R (the parameter R represents the maximum number of nodes that the reverse neighbor list rnn_new and rnn_old can contain), the normal storage operation is carried out; when the number of nodes is not less than R, whether the new node replaces the node already in the reverse neighbor list rnn_new or rnn_old is judged according to the generated random integer, so that the unnecessary node storage is greatly avoided, and the required memory is reduced.

[0055] 3. Compared with the NN-Descent method in the prior art, the neighbor list nn_new and nn_old structure is no longer used, and new flag bits flag (new' and old') are used to represent the nodes originally stored in the neighbor list nn_new and nn_old, so that the K nearest neighbor graph construction process does not need to use additional data structures, and the memory consumption is reduced.

[0056] 4. The application reduces the CPU cache miss times by dividing the reverse neighbor list rnn_new and rnn_old into blocks and processing each sub-block respectively, and improves the K nearest neighbor graph construction efficiency.

[0057] In summary, the application avoids the unnecessary node storage and the use of unnecessary data structures by quantizing the data, reduces the memory consumption, divides the reverse neighbor list rnn_new and rnn_old into blocks, reduces the cache miss times in the CPU, speeds up the data access efficiency, improves the K nearest neighbor graph construction efficiency on the basis of ensuring the high recall rate of the K nearest neighbor graph, and reduces the required memory when constructing the K nearest neighbor graph. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The flowchart of the application.

[0059] Figure 2 The flowchart of the Join operation of each node in the application.

[0060] Figure 3 The flowchart of the update operation of each node in the application.

[0061] Figure 4 The time required for the application to construct the K nearest neighbor graph on the Sift1M data set.

[0062] Figure 5 The time required for the application to construct the K nearest neighbor graph on the Gist data set.

[0063] Figure 6 Memory required for constructing K-Nearest Neighbor Graph on SiftlM dataset by the present application.

[0064] Figure 7 Memory required for constructing K-Nearest Neighbor Graph on Gist dataset by the present application. DETAILED DESCRIPTION

[0065] The technical solutions of the present application are further described in detail in combination with the drawings.

[0066] As shown in Figure 1 , a fast K-Nearest Neighbor Graph construction method comprises the following steps:

[0067] Step 1: reading a dataset D from a user-specified hard disk storage location into memory, and quantizing the dataset D;

[0068] reading a dataset D from a user-specified hard disk storage location into memory, the dataset D containing n nodes u in a d-dimensional space; in the dataset D, each node u is represented by a d-dimensional vector; according to the number n of nodes u, reading each vector in the dataset D for n times, and recording the maximum value max and the minimum value min of the nxd values in the entire dataset D; again reading each vector in the dataset D for n times, quantizing each value to obtain a new value, and the new values constitute a quantized dataset D, at this time, the quantized dataset D is saved in the memory, and the original dataset D is no longer in the memory, not occupying the memory space;

[0069] The quantization formula is as follows:

[0070]

[0071] wherein v represents the original value, nv represents the new value; max represents the maximum value of the nxd values in the dataset D, and min represents the minimum value of the nxd values in the dataset D;

[0072] Step 2: initializing the neighbor candidate set pool and the reverse neighbor list rnn_new of each node u in the quantized dataset D obtained in step 1;

[0073] randomly selecting S nodes u from the n nodes u of the quantized dataset D s , the selected nodes u s are different from the current node u i , respectively calculating the distance between the current node u and the S nodes u i , and arranging the S nodes u in the order from near to far to the current node u s i s ​​Sort the nodes to obtain the initialized neighbor candidate set pool. The flag bit of each node u in the initialized neighbor candidate set pool is in the state of new.

[0074] Randomly select 2S nodes u from the n nodes u of the quantized dataset D. s , 2S nodes u s Fill directly into the current node u i The initialized reverse neighbor list rnn_new is obtained from the reverse neighbor list rnn_new; the parameter S is used to balance the construction time and recall rate of the K-nearest neighbor graph.

[0075] Step 3, join operation: For each node u, calculate the distance between different nodes in the neighbor candidate set pool, the reverse neighbor lists rnn_new and rnn_old, and update the neighbor candidate set pool of the corresponding node in the reverse neighbor lists rnn_new and rnn_old based on the calculation results, such as... Figure 2 As shown, the specific process is as follows:

[0076] If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of new or old, then skip the current node u. i ;

[0077] If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of 'new', then the node u will be... i And arranged at node u i The flag following it indicates whether the state is 'new' or 'old', or whether the node u in the reverse neighbor lists rnn_new and rnn_old is new. j Perform distance calculations separately for node u. i and node u j After the distance between them is calculated, based on node u i and node u j The distance between them, using node u i Go update node u j The neighbor candidate set pool, while using node u j Go update node u i The neighbor candidate set pool; the updated neighbor candidate set pool of node u contains nodes u. i The flag flag is reset to old;

[0078] If a node u is in the candidate neighbor set pool of node u iIf the flag is in the state of 'old', then the node u will be... i And arranged at node u i The flag following the node is in the state of 'new' and the node u in the reverse neighbor list rnn_new. j Perform distance calculations separately for node u. i and node u j After the distance between them is calculated, based on node u i and node u j The distance between them, using node u i Go update node u j The neighbor candidate set pool, while using node u j Go update node u i The neighbor candidate set pool; the updated neighbor candidate set pool of node u contains nodes u. i The flag flag is reset to old;

[0079] Simultaneously, distance calculations are performed on different nodes in the reverse neighbor lists rnn_new and rnn_old of node u. Before the calculation, the reverse neighbor lists rnn_new and rnn_old of node u are first divided into blocks to obtain sub-blocks of the reverse neighbor lists rnn_new and rnn_old of node u, with the number of sub-blocks being respectively... Blocks and The sub-blocks are defined as follows: num(rnn_new) and num(rnn_old) represent the number of nodes in the reverse neighbor lists rnn_new and rnn_old of node u, respectively; NIOB represents the number of nodes stored in each sub-block; and the sub-blocks of the reverse neighbor list rnn_new are labeled N1, N2, ..., N. i The sub-blocks of the reverse neighbor list rnn_old are labeled O1, O2, ..., O j The formula for calculating the number of nodes NIOB stored in each sub-block is as follows:

[0080] NIOB=L2CacheSize / 2 / (dim*2+sizeof(Neighbor)) / 2

[0081] Where L2CacheSize represents the size of the L2 cache in the CPU, dim represents the dimension of the node vector, sizeof(Neighbor) represents the size of the space occupied by each Neighbor in the neighbor candidate set pool, and the Neighbor includes the id of the current node, the distance from the current node u to nodes different from the current node u, and the flag of the current node u.

[0082] Iterate through all sub-blocks of the reverse neighbor lists rnn_new and rnn_old, and for all nodes u in the two sub-blocks... i and node u j Perform distance calculation; node u i and node u j After the distance between them is calculated, based on node u i and node u j The distance between them, using node u i Go update node u j The neighbor candidate set pool, while using node u j Go update node u i The candidate set of neighbors, pool;

[0083] The two sub-blocks include the following cases:

[0084] (1) The same sub-blocks (N) in the reverse neighbor list rnn_new i With N i If the same sub-block (N) i With N i If the number of nodes in a given array is m, then the number of distance calculations is . Second-rate;

[0085] (2) Different sub-blocks of the reverse neighbor list rnn_new (N i1 With N i2 If different sub-blocks (N) i1 With N i2 If the number of nodes in the distance calculation is m and n respectively, then the distance calculation will be performed m × n times.

[0086] (3) Sub-blocks (N) of the reverse neighbor list rnn_new i ) and the sub-blocks of rnn_old (O j ), if sub-block (N i ) and sub-blocks (O) j If the number of nodes in the distance calculation is m and n respectively, then the distance calculation will be performed m × n times.

[0087] According to node u i and node u j The distance between them, using node u j Go update node u i The operation process of the neighbor candidate set pool and the use of node u i Go update node u j The operation process of the neighbor candidate set pool is the same; the use of node u j Go update node u i The specific process of pooling the neighbor candidate set is as follows:

[0088] First, it is judged whether node u j has been in the neighbor candidate set pool of node u i , if node u j has been in the neighbor candidate set pool of node u i , then node u j is skipped; if node u j is not in the neighbor candidate set pool of node u i , it is judged whether the number of nodes in the neighbor candidate set pool of node u i is less than L, wherein L is the maximum capacity of the neighbor candidate set pool;

[0089] if the number of nodes in the neighbor candidate set pool of node u i is less than L, then node u j is inserted into the neighbor candidate set pool of node u i in a proper position, so that the nodes in the neighbor candidate set pool of node u i are arranged from near to far to node u i , and the state of the flag of node u j is new;

[0090] if the number of nodes in the neighbor candidate set pool of node u i is not less than L, then the distance between node u i and node u j is compared with the distance between node u i and the last node in the neighbor candidate set pool of node u i ;

[0091] if the distance between node u i and the last node in the neighbor candidate set pool of node u i is greater than the distance between node u i and node u j , then the last node in the neighbor candidate set pool of node u i is deleted, and then node u j is inserted into the neighbor candidate set pool of node u i in a proper position, so that the nodes in the neighbor candidate set pool of node u i are arranged from near to far to node u i , and the state of the flag of node u j is new;

[0092] if the distance between node u i and the last node in the neighbor candidate set pool of node u ithe distance between the last node in the neighbor candidate set pool of node u i and node u j is not greater than the distance between node u j ;

[0093] Step 4: judging whether the recall rate set by the user or the number of times of performing the join operation is satisfied, if yes, the iteration ends, and the first K nodes u are selected from the neighbor candidate set pool of each node u in the last iteration of the data set D K , to obtain the K-neighbor graph of the data set D; if no, the next step is entered; the parameter K is used to control the out-degree of the K-neighbor graph, i.e., the number of neighbors to be searched by each node;

[0094] Step 5, update operation: updating the flag of each node in the neighbor candidate set pool of each node u in the data set D and the reverse neighbor list rnn_new and rnn_old, after traversing each node u in the data set D, returning to step 3, as shown in Figure 3 ;

[0095] For each node u in the data set D in step 2, the reverse neighbor list rnn_new and rnn_old of node u are emptied, and the variable M is updated according to the parameter S in step 2 and the flag state of each node u in the neighbor candidate set pool in step 3, so that at most S nodes marked as new are included in the first M nodes in the neighbor candidate set pool in step 3, which can balance the efficiency and recall rate of constructing the K-neighbor graph;

[0096] For the first M nodes u in the neighbor candidate set pool in step 3 M , if the flag state of the current node u M is new, the flag state of the current node u M is reset to new', and the node u of the data set D is stored in the reverse neighbor list rnn_new of the corresponding node u M ; if the flag state of the current node u M is old, the flag state of the current node u M is reset to old', and the node u of the data set D is stored in the reverse neighbor list rnn_old of the corresponding node u M ; until the first M nodes u in the neighbor candidate set pool of node u are traversed; after traversing each node u in the data set D, returning to step 3;

[0097] The specific process of storing is as follows:

[0098] judging whether the current node u Mthe number of nodes in the reverse neighbor list rnn_new or rnn_old of the node u and the parameter R, if the number of nodes is less than R, the node u is directly inserted into the current node u M the last of the reverse neighbor list rnn_new or rnn_old of the node u; if the number of nodes is not less than R, a random integer rand is obtained, the random integer rand is 0 to the execution of the storage of the current node u M the number of times of the reverse neighbor list rnn_new or rnn_old of the node u; if the random integer rand is less than R, the node u is replaced by the current node u M the randth node of the reverse neighbor list rnn_new or rnn_old of the node u; the parameter R is used to represent the maximum number of nodes that can be contained in the reverse neighbor list rnn_new and rnn_old.

[0099] A K-neighbor graph construction system, comprising:

[0100] A quantization module: for quantizing the data set D read from the user-specified hard disk storage location into memory;

[0101] An initialization module: for initializing the neighbor candidate set pool and the reverse neighbor list rnn_new of each node u in the quantized data set D;

[0102] A join module: for calculating the distance between different nodes of the neighbor candidate set pool, the reverse neighbor list rnn_new and rnn_old of each node u, and updating the neighbor candidate set pool of the corresponding node of the neighbor candidate set pool, the reverse neighbor list rnn_new and rnn_old of the node u according to the calculation result;

[0103] A judgment module: for judging whether the recall rate set by the user or the number of times of performing the join operation is satisfied, if yes, the iteration is ended, and the first K nodes u K are selected from the neighbor candidate set pool of each node u in the last iteration, and the K-neighbor graph of the data set D is obtained; if not, the update module is entered;

[0104] An update module: for updating the flag of each node in the neighbor candidate set pool of each node u and the reverse neighbor list rnn_new and rnn_old, and returning to the join module after traversing each node u in the data set D.

[0105] A K-neighbor graph construction device, comprising:

[0106] A memory: for storing the computer program of the K-neighbor graph construction method described above, which is a computer-readable device;

[0107] Processor: for executing the computer program to realize the above-mentioned K-neighbor graph construction method.

[0108] A computer readable storage medium stores a computer program, which can realize the above-mentioned K-neighbor graph construction method when executed by a processor.

[0109] The data set D used in the embodiment is Sift1M data set and Gist data set, which are commonly used high-dimensional data sets from the real world, Sift1M contains one million 128-dimensional vectors, and Gist contains one million 960-dimensional vectors, wherein the parameter K is 20; the existing method NN-Descent is used to construct a K-neighbor graph as a control group, and the beneficial effects of the K-neighbor graph construction method of the present application are further verified. Figure 4 and Figure 5 The horizontal axis represents time, and the vertical axis represents the recall of the constructed K-neighbor graph. Figure 6 and Figure 7 The horizontal axis represents the number of iterations, and the vertical axis represents the memory required for the constructed K-neighbor graph.

[0110] Figure 4 is the time required for the existing method NN-Descent and the K-neighbor graph construction method of the present application to construct a K-neighbor graph on the Sift1M data set, respectively. Figure 4 It can be seen that, compared with the existing method NN-Descent, the method of the present application can significantly reduce the time required for constructing a K-neighbor graph with the same recall on the Sift1M data set, i.e., it can reduce the time overhead by 18%, greatly improving the speed of the construction method.

[0111] Figure 5 is the time required for the existing method NN-Descent and the K-neighbor graph construction method of the present application to construct a K-neighbor graph on the Gist data set, respectively. Figure 5 It can be seen that, compared with the existing method NN-Descent, the method of the present application can significantly reduce the time required for constructing a K-neighbor graph with the same recall on the Gist data set, i.e., it can reduce the time overhead by 31%, greatly improving the speed of the construction method.

[0112] Figure 6 is the memory required for the existing method NN-Descent and the K-neighbor graph construction method of the present application to construct a K-neighbor graph on the Sift1M data set, respectively. Figure 6It can be seen that, compared with the existing method NN-Descent, the method of the present application can significantly reduce the memory required for building the K-Nearest Neighbor graph with the same recall rate on the Sift1M dataset, i.e. can reduce the memory overhead by 29%.

[0113] Figure 7 The memory required for building the K-Nearest Neighbor graph using the existing method NN-Descent and the method of the present application respectively on the Gist dataset. It can be seen that, compared with the existing method NN-Descent, the method of the present application can significantly reduce the memory required for building the K-Nearest Neighbor graph with the same recall rate on the Gist dataset, i.e. can reduce the memory overhead by 42%. Figure 7

[0114] In summary, compared with the existing method NN-Descent, the method of the present application can reduce the time overhead by 18% and 31% respectively on the Sift1M and Gist datasets for building the K-Nearest Neighbor graph with the same recall rate; can reduce the memory overhead by 29% and 42% respectively on the Sift1M and Gist datasets, reduce the memory required in the process of building the K-Nearest Neighbor graph, and build the K-Nearest Neighbor graph at a faster speed.​

Claims

1. A method for constructing a K-nearest neighbor graph, characterized in that, Includes the following steps: Step 1: Read dataset D from the user-specified hard disk storage location into memory and quantize dataset D; Step 2: Initialize the neighbor candidate set pool and the reverse neighbor list rnn_new for each node u in the quantized dataset D obtained in Step 1; Step 3, join operation: For each node u, calculate the distance between different nodes in the neighbor candidate set pool, the reverse neighbor lists rnn_new and rnn_old, and update the neighbor candidate set pool of the corresponding node in the reverse neighbor lists rnn_new and rnn_old based on the calculation results; the specific process is as follows: If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of new or old, then skip the current node u. i ; If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of 'new', then the node u will be... i And arranged at node u i The flag following it indicates whether the state is 'new' or 'old', or whether the node u in the reverse neighbor lists rnn_new and rnn_old is new. j Perform distance calculations separately; based on node u i and node u j The distance between them, using node u i Go update node u j The neighbor candidate set pool, while using node u j Go update node u i The candidate set of neighbors, pool; The updated neighbor candidate set pool contains nodes u. i The flag flag is reset to old; If a node u is in the candidate neighbor set pool of node u i If the flag is in the state of 'old', then the node u will be... i And arranged at node u i The flag following the node is in the state of 'new' and the node u in the reverse neighbor list rnn_new. j Perform distance calculations separately; based on node u i and node u j The distance between them, using node u i Go update node u j The neighbor candidate set pool, while using node u j Go update node u i The candidate set of neighbors, pool; The updated neighbor candidate set pool contains nodes u. i The flag flag is reset to old; Simultaneously, distance calculations are performed on different nodes in the reverse neighbor lists rnn_new and rnn_old of node u. Before the calculation, the reverse neighbor lists rnn_new and rnn_old of node u are first divided into blocks to obtain sub-blocks of the reverse neighbor lists rnn_new and rnn_old of node u, with the number of sub-blocks being respectively... Blocks and The sub-blocks are defined as follows: num(rnn_new) and num(rnn_old) represent the number of nodes in the reverse neighbor lists rnn_new and rnn_old of node u, respectively; NIOB represents the number of nodes stored in each sub-block; and the sub-blocks of the reverse neighbor list rnn_new are labeled N1, N2, ..., N. i The sub-blocks of the reverse neighbor list rnn_old are labeled O1, O2, ..., O j The formula for calculating the number of nodes NIOB stored in each sub-block is as follows: NIOB=L2CacheSize / 2 / (dim*2+sizeof(Neighbor)) / 2 Where L2CacheSize represents the size of the L2 cache in the CPU, dim represents the dimension of the node vector, sizeof(Neighbor) represents the size of the space occupied by each Neighbor in the neighbor candidate set pool, and the Neighbor includes the id of the current node, the distance from the current node u to nodes different from the current node u, and the flag of the current node u. Iterate through all sub-blocks of the reverse neighbor lists rnn_new and rnn_old, and for all nodes u in the two sub-blocks... i and node u j Perform distance calculation; based on node u i and node u j The distance between them, using node u i Go update node u j The neighbor candidate set pool, while using node u j Go update node u i The candidate set of neighbors, pool; The two sub-blocks include the following cases: (1) The same sub-blocks (N) in the reverse neighbor list rnn_new i With N i If the same sub-block (N) i With N i If the number of nodes in a given array is m, then the number of distance calculations is . Second-rate; (2) Different sub-blocks of the reverse neighbor list rnn_new (N i1 With N i2 If different sub-blocks (N) i1 With N i2 If the number of nodes in the distance calculation is m and n respectively, then the distance calculation will be performed m × n times. (3) Sub-blocks (N) of the reverse neighbor list rnn_new i ) and the sub-blocks of rnn_old (O j ), if sub-block (N i ) and sub-blocks (O) j If the number of nodes in the distance calculation is m and n respectively, then the distance calculation will be performed m × n times. Step 4: Determine if the user-defined recall rate or the number of join operations is met. If so, the iteration ends, and the top K nodes u are selected from the candidate neighbor set pool of each node u in the last iteration. K If not, proceed to the next step; the parameter K controls the out-degree of the K-nearest neighbor graph, that is, the number of neighbors that each node needs to find. Step 5, update operation: Update the flag of each node in the neighbor candidate set pool of each node u, as well as the reverse neighbor lists rnn_new and rnn_old. After traversing each node u in the dataset D, return to step 3.

2. The method for constructing a K-nearest neighbor graph according to claim 1, characterized in that, The specific process of step 1 is as follows: Dataset D is read from the user-specified hard disk storage location into memory. Dataset D contains n nodes u in d-dimensional space. In dataset D, each node u is represented by a d-dimensional vector. Based on the number of nodes u n, each vector in dataset D is read n times, and the maximum value max and minimum value min among the n×d values ​​of the entire dataset D are recorded. Each vector in dataset D is read n times again, and each value is quantized to obtain new values. These new values ​​constitute the quantized dataset D, which is stored in memory. The quantization formula is as follows: Where v represents the original value, nv represents the new value; max represents the maximum value of n×d values ​​in dataset D, and min represents the minimum value of n×d values ​​in dataset D.

3. The method for constructing a K-nearest neighbor graph according to claim 1, characterized in that, The specific process of step 2 is as follows: Randomly select S nodes u from the n nodes u of the quantized dataset D. s The selected node u s Unlike the current node u i Calculate the current node u respectively i With S nodes u s The distances between the S nodes are calculated in ascending order from the nearest to the farthest. s Sort the nodes to obtain an initialized neighbor candidate set pool. The flag bit of each node u in the initialized neighbor candidate set pool is in the state of new. Randomly select 2S nodes u from the n nodes u of the quantized dataset D. s , 2S nodes u s Fill directly into the current node u i From the reverse neighbor list rnn_new, we obtain the initialized reverse neighbor list rnn_new.

4. The method for constructing a K-nearest neighbor graph according to claim 1, characterized in that, In step 3, according to node u i and node u j The distance between them, using node u j Go update node u i The operation process of the neighbor candidate set pool and the use of node u i Go update node u j The operation process of the neighbor candidate set pool is the same; the use of node u j Go update node u i The specific process of pooling the candidate neighbor set is as follows: First, determine node u. j Is it already at node u? i In the neighbor candidate set pool, if node u j Already at node u i If the neighbor candidate set pool is used, then skip node u. j If node u j Not in node u i In the neighbor candidate set pool, determine node u i Whether the number of nodes in the neighbor candidate set pool is less than L, where L is the maximum capacity of the neighbor candidate set pool; If node u i If the number of nodes in the neighbor candidate set pool is less than L, then node u will be added to the list. j Insert into u i The appropriate position of node u in the neighbor candidate set pool. i The nodes in the neighbor candidate set pool are ranked according to their distance from node u. i Arranged from nearest to farthest, node u j The flag is in a new state; If node u i If the number of nodes in the neighbor candidate set pool is not less than L, then the comparison node u i and node u j Distance between nodes u i and node u i The relationship between the distances between the last nodes in the neighbor candidate set pool; If node u i and node u i In the neighbor candidate set pool, the distance between the last node is greater than that between nodes u. i and node u j If the distance between them is zero, then delete node u. i The node that is last in the neighbor candidate set pool, and then node u. j Insert into node u i The appropriate position of node u in the neighbor candidate set pool. i The nodes in the neighbor candidate set pool are ranked according to their distance from node u. i Arranged from nearest to farthest, node u j The flag is in a new state; If node u i and node u i The distance between the last nodes in the neighbor candidate set pool is no greater than that between nodes u. i and node u j If the distance between them is such that node u is skipped directly. j .

5. The method for constructing a K-nearest neighbor graph according to claim 1, characterized in that, The specific process of step 5 is as follows: For each node u in the dataset D in step 2, clear the reverse neighbor list rnn_new and rnn_old of node u, and update the variable M according to the parameter S in step 2 and the flag status of each node u in the neighbor candidate set pool in step 3, so that at most S nodes in the first M nodes of the neighbor candidate set pool in step 3 are marked as new. For the first M nodes u of the neighbor candidate set pool in step 3 M If the current node u M If the flag bit state is "new", then reset the flag bit state to "new'" and store the corresponding node u from dataset D into the corresponding node u. M In the reverse neighbor list rnn_new; if the current node u M If the flag state is 'old', then reset the flag state to 'old' and store node u from dataset D into the corresponding node u. M The reverse neighbor list rnn_old; until the first M nodes u in the neighbor candidate set pool of node u have been traversed. M After traversing each node u in the dataset D, return to step 3.

6. The method for constructing a K-nearest neighbor graph according to claim 5, characterized in that, The specific process for storing data in step 5 is as follows: Determine the current node u M The relationship between the number of nodes in the reverse neighbor list rnn_new or rnn_old and the parameter R is determined. If the number of nodes is less than R, then node u is directly inserted into the current node u. M The last node in the reverse neighbor list rnn_new or rnn_old; if the number of nodes is not less than R, a random integer rand is obtained, and from the random integer rand being 0 to the node u being stored in the current node. M The number of times the reverse neighbor list rnn_new or rnn_old is counted; If the random integer rand is less than R, then replace the current node u with node u. M The reverse neighbor list rnn_new or rnn_old contains the rand-th node; the parameter R is used to represent the maximum number of nodes that the reverse neighbor lists rnn_new and rnn_old can contain.

7. A system for constructing a K-nearest neighbor graph according to any one of claims 1-6, characterized in that, include: Quantization module: Used to quantize the dataset D read from the user-specified hard drive storage location into memory; Initialization module: Used to initialize the neighbor candidate set pool and reverse neighbor list rnn_new for each node u in the quantized dataset D; The join module is used to calculate the distance between different nodes in the neighbor candidate set pool, the reverse neighbor list rnn_new and rnn_old for each node u, and update the neighbor candidate set pool of the corresponding node in the reverse neighbor list rnn_new and rnn_old based on the calculation results. The decision module determines whether the user-defined recall rate or the number of join operations is met. If so, the iteration ends, and the top K nodes u are selected from the neighbor candidate set pool of each node u in the last iteration. K If not, obtain the K-nearest neighbor graph of dataset D; otherwise, proceed to the update module. The update module is used to update the flag of each node in the neighbor candidate set pool of each node u, as well as the reverse neighbor lists rnn_new and rnn_old. After traversing each node u in the dataset D, it returns to the join module.

8. A device for constructing a K-nearest neighbor graph, characterized in that, include: Memory: A computer program storing a method for constructing a K-nearest neighbor graph as described in any one of claims 1-6, and is a computer-readable device; Processor: configured to implement the method for constructing a K-nearest neighbor graph according to any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of a method for constructing a K-nearest neighbor graph as described in any one of claims 1-6.

Images