Data indexing method and system

Abstract

The application provides a data indexing method and system, the method comprising: determining a first index value of to-be-matched data in a sample set according to a preset indexing construction method; and determining a first preset number of target data most matched with the to-be-matched data according to an index table of an original data set in the sample set and the first index value. The system is used for executing the above method. The data indexing method and system provided by the application combine a vector quantization method and a BIRCH algorithm idea, divide the sample set into a plurality of sub-vectors first, establish a clustering feature tree for each group of sub-vectors by using the BIRCH algorithm, and record index values of each leaf node, so that the target data most matched with the to-be-matched data can be quickly retrieved from the index table of the original data set based on the index value of the to-be-matched data, and the data matching speed is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data indexing method and system. BACKGROUND

[0002] High-dimensional large data set indexing is a problem widely existing in the fields of artificial intelligence, big data, information retrieval, etc. For example, when performing image recognition and searching in the field of artificial intelligence, image data is often converted into high-dimensional vectors and then searched and matched in a large number of image sets. For another example, when performing text matching in natural language processing, texts are also converted into high-dimensional vectors such as word vectors and text vectors and then matched with a large number of data in a text library. In these scenarios, if a method of sequentially calculating similarity between to-be-matched data and data sets and then sorting is used, the query efficiency is low, which affects the normal operation of production business.

[0003] The methods in the prior art for solving the problem of high-dimensional large data set indexing, such as Annoy (Approximate Nearest Neighbors Oh Yeah) algorithm and Local Sensitive Hashing (LSH) method, can provide high-dimensional vector data indexing capability to a certain extent, but once the indexing model is established, new indexing data cannot be added, otherwise the indexing model needs to be retrained, which will consume a large amount of computing resources in the big data scenario. The vector quantization method such as Product Quantization maps high-dimensional vectors to a small number of codebooks, but still needs to calculate the similarity of all to-be-matched data when searching for similar data. For example, if high-dimensional vectors are mapped to M codebooks and the amount of to-be-matched data is N, the search time complexity is O(MN). When the amount of data is large and the value of N is large, the matching algorithm will consume a large amount of time for calculation, which is difficult to meet the actual production requirements of fast indexing.

[0004] By minimizing the loss function, the final binary coding matrix and the binary coding weighted by the final weight matrix are obtained. When matching, the binary coding of the full amount of data processed by the same method in the database needs to be compared, and the weighted Hamming distance is calculated. When the amount of data is large, it is difficult to ensure the matching speed.

[0005] The K-means clustering method is used to calculate the codebook of data. After obtaining the codebook, the similarity of the full amount of data also needs to be calculated, so the improvement of retrieval efficiency is also very limited. SUMMARY

[0006] The data index method and system provided by the application are used to solve at least one of the above problems in the prior art, and the sample set is first divided into multiple sub-vectors by combining a vector quantization method and a BIRCH algorithm idea, a clustering feature tree is established for each group of sub-vectors by using the BIRCH algorithm, and index values of each leaf node are recorded, based on the index value of the to-be-matched data, the most matched target data of the to-be-matched data can be quickly retrieved from the index table of the original data set, and the data matching speed is improved.

[0007] The data index method provided by the application comprises:

[0008] According to the preset index construction method, a first index value of to-be-matched data in the sample set is determined.

[0009] According to the index table of the original data set in the sample set and the first index value, a first preset number of target data most matched with the to-be-matched data are determined.

[0010] The index table is determined according to a second index value of the original data set.

[0011] The second index value is determined according to the index value of the original data set obtained according to the preset index construction method.

[0012] The preset index construction method comprises:

[0013] The sample set is divided into a second preset number of sub-vectors, and each sub-vector is clustered based on a BRICH algorithm to obtain a clustering feature tree of each sub-vector.

[0014] According to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree, an index value of each sub-vector is determined.

[0015] According to the index value of each sub-vector, an index value of the sample set is determined.

[0016] According to the data index method provided by the application, the index value of each sub-vector is determined according to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree, and the method comprises:

[0017] The clustering feature tree is traversed, and the index point value of each node of all layers in the clustering feature tree is determined according to the depth and the maximum number of nodes.

[0018] The sample set is traversed, and the index value of the most adjacent node of each sub-vector in the current layer is determined according to the index point value, and the index value of each sub-vector is determined by traversing each layer.

[0019] According to the data indexing method provided by the application, the first preset number of target data most matched with the to-be-matched data are determined according to the index table of the original data set in the sample set and the first index value, and the method comprises the following steps:

[0020] If the first index value of the sub-vector of the to-be-matched data and the second index value of the sub-vector of the original data set in the same level are the same, it is determined that the sub-vector of the to-be-matched data and the sub-vector of the original data set are matched;

[0021] According to the sub-vector of the original data set matched with the to-be-matched data, the matched data are determined;

[0022] If the number of the matched data is greater than or equal to the preset number, the first preset number of target data most matched with the to-be-matched data are determined according to the similarity between the to-be-matched data and the matched data.

[0023] According to the data indexing method provided by the application, the index table is updated, and the method comprises the following steps:

[0024] According to the preset index construction method, the index value of the to-be-added data in the sample set is determined;

[0025] According to the index value of the to-be-added data, the first target leaf node to which each sub-vector of the to-be-added data belongs is determined in each clustering feature tree of the original data set;

[0026] If the difference between the number of clustering feature nodes contained in the first target leaf node and the preset maximum clustering number is greater than a first preset value, the first target leaf node and the target parent node are split based on the preset index construction method;

[0027] The index value of the split target sub-node is added to the index table;

[0028] The first related data in the original data set affected is determined, and the second index value of the first related data is updated based on the preset index construction method;

[0029] The target parent node is determined according to the parent node of the first target leaf node.

[0030] According to the data indexing method provided by the application, the index value of the split target sub-node is added to the index table, and the method comprises the following steps:

[0031] If the number of split sub-nodes is less than the preset maximum sub-node threshold, the split target sub-node is directly split, and the index value of the split target sub-node is added to the index table;

[0032] If the number of the split sub-nodes is greater than the preset maximum sub-node threshold, the target sub-node is split.

[0033] According to the depth of the cluster feature tree of the split original data set and the number of nodes of each layer of the cluster feature tree of the split original data set, the index value of the split target sub-node is determined, and the index value of the split target sub-node is added to the index table.

[0034] According to the data indexing method provided by the application, the index table is updated, and further comprising:

[0035] The index value of the to-be-deleted data in the original data set is obtained.

[0036] According to the index value of the to-be-deleted data, the second target leaf node to which each sub-vector of the to-be-deleted data belongs in each cluster feature tree of the original data set is determined.

[0037] The relationship between the to-be-deleted data and the second target leaf node in each cluster feature tree is deleted, and node contraction judgment is performed.

[0038] If the number of contracted nodes is greater than the second preset value, the target node closest to the second target leaf node is determined.

[0039] If the sum of the number of cluster feature nodes contained in the target node and the number of cluster feature nodes contained in the second target leaf node is less than the preset maximum clustering number, the index value of the target node is updated according to the index value of the target node and the index value of the second target leaf node.

[0040] The second related data in the original data set affected is determined, and the second index value of the second related data is updated based on the preset index construction method.

[0041] According to the data indexing method provided by the application, the index value, comprising:

[0042] Index identification and the number of bits of index identification.

[0043] The application also provides a data indexing system, comprising: a data indexing module and a data matching module;

[0044] The data indexing module is configured to determine the first index value of the to-be-matched data in the sample set according to a preset index construction method.

[0045] The data matching module is configured to determine the first preset number of target data most matched with the to-be-matched data according to the index table of the original data set in the sample set and the first index value.

[0046] wherein the index table is determined according to a second index value of the original data set;

[0047] The second index value is determined by an index value of the original data set obtained according to the preset index construction method;

[0048] The data index module comprises an index construction submodule, configured to divide the sample set into a second preset number of sub-vectors, and perform clustering on each sub-vector based on a BIRCH algorithm to obtain a clustering feature tree of each sub-vector;

[0049] An index value of each sub-vector is determined according to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree;

[0050] An index value of the sample set is determined according to the index value of each sub-vector.

[0051] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data index method according to any one of the above when executing the program.

[0052] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the data index method according to any one of the above.

[0053] The data index method and system provided by the application combine the vector quantization method and the BIRCH algorithm idea, divide the sample set into multiple sub-vectors first, use the BIRCH algorithm to establish a clustering feature tree for each group of sub-vectors, and record the index value of each leaf node, so that the most matched target data of the to-be-matched data can be quickly retrieved from the index table of the original data set based on the index value of the to-be-matched data, and the data matching speed is improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0055] Figure 1 is a flowchart of the data index method provided by the application;

[0056] Figure 2 is an example schematic diagram of the index value of the original data set provided by the application;

[0057] Figure 3 is a schematic diagram of an indexing process provided by the present application;

[0058] Figure 4 is a schematic diagram of an index table updating process provided by the present application;

[0059] Figure 5 is a schematic diagram before splitting provided by the present application;

[0060] Figure 6 is a schematic diagram after splitting provided by the present application;

[0061] Figure 7 is a schematic diagram of a data indexing system provided by the present application;

[0062] Figure 8 is a schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0064] Figure 1 is a schematic diagram of a data indexing method provided by the present application, as shown in Figure 1 the method comprises:

[0065] S1, determining a first index value of to-be-matched data in a sample set according to a preset index construction method;

[0066] S2, determining a first preset number of target data most matched with the to-be-matched data according to an index table of an original data set in the sample set and the first index value;

[0067] wherein the index table is determined according to a second index value of the original data set;

[0068] The second index value is determined by an index value of the original data set obtained according to the preset index construction method;

[0069] The preset index construction method comprises:

[0070] S11, dividing the sample set into a second preset number of sub-vectors, and clustering each sub-vector based on a BRICH algorithm to obtain a clustering feature tree of each sub-vector;

[0071] S12, determine the index value of each sub-vector according to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree;

[0072] S13, determine the index value of the sample set according to the index value of each sub-vector. It should be noted that the execution subject of the above method can be a computer device.

[0073] Further, in one embodiment, the index value can specifically include:

[0074] The index identifier and the number of bits of the index identifier.

[0075] Optionally, the sample set to be processed can be specifically the original data set X or the data to be matched wx, the sample set (such as X or wx) is divided into a second preset number of sub-vectors, each sub-vector is clustered based on the BRICH algorithm, and the corresponding clustering feature tree (CFTREE) of each sub-vector is obtained. According to the depth of the CFTREE and the maximum number of nodes of each layer of the CFTREE, the index value of each sub-vector is obtained, and finally the index values of each sub-vector are spliced to obtain the index value of the sample set.

[0076] The above index value construction method of the sample set can be used as a preset index construction method to obtain the index value of the data to be matched wx in the sample set, that is, the first index value.

[0077] According to the preset index construction method, the original data set X in the sample set can be processed to obtain the index value of the original data set, that is, the second index value, and the index table of the original data set is constructed according to the second index value.

[0078] According to the first index value and the index table, the first preset number of target data in the original data set X that are most matched with the data to be matched wx are obtained.

[0079] Wherein, the basic concept of BRICH algorithm is shown in Table 1:

[0080] Table 1

[0081]

[0082] Wherein, N is the number of nodes in the subclass, is the linear sum of N nodes, SS is the square sum of N nodes, N1 is the number of nodes in the first subclass, is the linear sum of N1 nodes, SS1 is the square sum of N1 nodes, N2 is the number of nodes in the second subclass, is the linear sum of N2 nodes, SS2 is the square sum of N2 nodes.

[0083] The data indexing method provided by the application combines a vector quantization method and a BIRCH algorithm thought, first divides a sample set into multiple sub-vectors, uses the BIRCH algorithm to establish a clustering feature tree for each group of sub-vectors, and records index values of each leaf node, based on the index values of the to-be-matched data, can quickly retrieve the target data most matched with the to-be-matched data from the index table of the original data set, and improves the data matching speed.

[0084] Further, in one embodiment, step S12 can specifically include:

[0085] S121, traversing the clustering feature tree, and determining the index point values of all nodes of each level in the clustering feature tree according to the depth and the maximum number of nodes;

[0086] S122, traversing the sample set, determining the index values of each sub-vector and the nearest node of the current level according to the index point values, and traversing each level to determine the index values of each sub-vector.

[0087] Optionally, assuming that the number of samples (x1, x2,..., x N ) in X is N, the sample dimension is Q, and the entire original data set X can be represented as an N×Q vector, that is:

[0088]

[0089] The Q-dimensional vector is divided into M groups of sub-vectors Let D=Q / M, and the dimension of each group of sub-vectors is N×Q, and it can be known that:

[0090]

[0091]

[0092] vstack is a matrix that merges the input matrices in the horizontal direction.

[0093] Based on the BIRCH algorithm, the expected number of leaf nodes LN and the maximum number of child nodes MCN of each node are set, and then the maximum number of samples T=N / LN contained in each cluster is obtained. The BIRCH clustering is performed on each sub-vector to obtain the CFTREE corresponding to each group of sub-vectors, that is Suppose that the CFTREE m contains CN clustering features (CF) nodes, and the leaf nodes are If is a leaf node , it is recorded as If is not a leaf node, the parent node of is recorded as

[0094] Step A0, constructing index table, if CFTREE m Depth is DP m , root node dp = 0, then from dp = 1 to construct the index table of CFTREE m , the maximum number of nodes of each layer of CFTREE m Determine the construction of BN m Bit index value Where, b m ∈ [0, 9], b is the index point value.

[0095] Step A1, traversing CFTREE m , determine the index value starting from dp t = 1 (such as dp = 1 layer has two, respectively ), then use To represent the index value of this level, the number of bits of the index value of this level is 1, get That is, determine The value range, so as to determine the index point value of each node of CFTREE m Each layer, recorded as

[0096] Step A2, let dp t+1 = dp t + 1, get Repeat the step until all levels of each node index point value is obtained.

[0097] Step A3, traversing From dp = 1 to determine the sub vector, for example The index value of the nearest node of the current layer If Then Traverse each level can be spliced to get the index value of the sub vector, for example Bit m,i :

[0098]

[0099] Step A4, repeat steps A0 to A3, can get the index value of each sub vector, through splicing get the index table B of the original data set X.

[0100]

[0101] Similarly, the index value of each sub vector in the data to be matched can be obtained.

[0102] For example Figure 2 ​​As shown, it is an example schematic diagram of the index value of the original data set provided by the application, which is obtained by Figure 2 It can be seen that the index point values of the first layer dp=1 nodes are b=1 and b=2 respectively, the index point values of the second layer dp=2 nodes are b=1, b=2, b=3 and b=4 respectively, and the index values of the nodes in dp=2 are bit=11, bit=12, bit=23 and bit=24 respectively.

[0103] The data indexing method provided by the application combines the high-dimensional large data set indexing construction method of the vector quantization method and the BIRCH algorithm idea, and in the indexing establishment process, the data set is divided into a plurality of sub-vectors, the CFTREE of each group of sub-vectors is established by using the BIRCH algorithm, and the sample data pointed by each leaf node is recorded, so that the indexing table construction of high-dimensional data is realized.

[0104] Further, in one embodiment, step S2 can specifically include:

[0105] S21, if the first index value of the sub-vector of the same level of the to-be-matched data is the same as the second index value of the sub-vector of the original data set, it is determined that the sub-vector of the to-be-matched data and the sub-vector of the original data set are matched;

[0106] S22, determining the matching data according to the sub-vector of the original data set matched with the to-be-matched data;

[0107] S23, if the number of matching data is greater than or equal to a preset number, determining the first preset number of target data most matched with the to-be-matched data according to the similarity between the to-be-matched data and the matching data.

[0108] Optionally, as shown, assuming that the dimension of the to-be-matched data wx is Q, according to the preset indexing construction method, it is divided into M sub-vectors Figure 3 Need to retrieve the first preset number of target data, for example, K target data, which are most matched with the to-be-matched data wx, that is, TOP-K data matching of the to-be-matched data wx:

[0109]

[0110] The segments of wx obtained after segmentation (segment 1, that is to segment M, that is ), the BIRCH model corresponding to each segment of wx (that is, BIRCH model 1 to BIRCH model M) is established according to the indexing construction method, and the index segment (that is, the index point value) of each sub-vector is output, such as index segment 1 to index segment M, so as to obtain the index value wbi of each sub-vector output by each BIRCH model:

[0111] ​

[0112] wherein wbi m is wx and CFREE m the index value of the nearest neighbor of each layer,

[0113] i.e.

[0114] Based on the index table B, match wbi:

[0115] define wherein mdp m represents that wx can match the index value of dp≤mdp m If the first index value of the sub-vector of the to-be-matched data of the same layer and the second index value of the sub-vector of the original data set are the same, i.e., when has or If then has or

[0116] initial mdp m , let mdp m = DP m , calculate Under the condition, the number of matched data samples MXN t ;

[0117] If MXN t ≥ K, then the to-be-matched data wx is sequentially calculated with the matched data to obtain the TOP-K similar data and the corresponding similarity after sorting.

[0118] If MXN t <K, take a random number rm, rm m＝rm = mdp m＝rm -1, calculate Under the condition, the number of matched MXN t+1 . If MXN t+1 <K, repeat the process; if MXN t+1 ≥ K, then the to-be-matched data wx is sequentially calculated with the matched data to obtain the first preset number of target data, for example, K target data, and the corresponding similarity after sorting.

[0119] The data indexing method provided by the application determines the data of the CF node to which the M sub-vectors of the to-be-matched data belong as similar data when indexing retrieval is performed, and automatically obtains the parent node information in the tree node according to the index vector when the amount of similar data is insufficient, thereby expanding the search range.

[0120] Further, in one embodiment, updating the index table can specifically include:

[0121] According to the preset index construction method, an index value of the to-be-added data in the sample set is determined;

[0122] According to the index value of the to-be-added data, a first target leaf node to which each sub-vector of the to-be-added data belongs is determined.

[0123] If a difference between the number of clustering feature nodes contained in the first target leaf node and the preset maximum clustering number is greater than a first preset value, the first target leaf node and a target parent node are split based on the preset index construction method;

[0124] The index value of the split target sub-node is added to the index table;

[0125] First related data in the original data set is determined, and a second index value of the first related data is updated based on the preset index construction method;

[0126] The target parent node is determined according to the parent node of the first target leaf node.

[0127] Optionally, Figure 4 is an index table updating process schematic diagram provided by the application, as Figure 4 shown:

[0128] Step B0, the dimension of the to-be-added data nx is Q, which is divided into M sub-vectors nxv m , m = 1, 2, …, M;

[0129]

[0130] Step B1, the segments (segment 1, i.e. to segment M, i.e. ) of nx obtained after the division are used to establish the BIRCH model corresponding to each segment of nx (i.e. BIRCH model 1 to BIRCH model M) according to the BRICH algorithm, and the index segment (i.e. index point value) of each sub-vector (segment of nx) is output, such as index segment 1 to index segment M, so as to obtain the index value nbi of each sub-vector output by each BIRCH model:

[0131]

[0132] Step B2, according to the index value nbi, the first target leaf node to which nx belongs in each CFTREE m is determined, which is assumed to be i.e. if , then

[0133] Step B3, let m'=m+1, assuming that the initial value of m is set to 1, determine If the difference between the number of CF nodes contained in the first target leaf node and the preset maximum clustering number B is greater than the first preset value (-1), that is, if CN-B>-1, the same method as in the preset index construction method is used to split the node and the parent node of the node, and the CF values of all parent nodes are updated:

[0134] Add the index value of the split target child node to the index table.

[0135] Determine the first related data in the affected original data set, and reconstruct the index value of the first related data to update the index value of the first related data.

[0136] Step B4, repeat step B3 until M CFTREE index update operations are completed.

[0137] The data index method provided by the application can realize the dynamic modification capability of adding data index, and can ensure data distribution and ensure the accuracy of the index when modifying the index.

[0138] Further, in one embodiment, adding the index value of the split target child node to the index table can specifically include:

[0139] If the number of split child nodes is less than the preset maximum child node threshold, directly split, and add the index value of the split target child node to the index table;

[0140] If the number of split child nodes is greater than the preset maximum child node threshold, split the target child node;

[0141] According to the depth of the cluster feature tree of the split original data set and the number of nodes of each layer of the cluster feature tree of the split original data set, determine the index value of the split target child node, and add the index value of the split target child node to the index table.

[0142] Optionally, if the number of split target child nodes is less than the preset maximum child node threshold MCN, directly split, and add the index value of the split target child node to the index table;

[0143] If the number of split target child nodes is greater than MCN, the target child node needs to be split to obtain target child nodes and At this time, the depth of the CFTREE m increases, Peer node is If After splitting If data Before splitting After splitting * indicates that the index bit can match any index identifier. The CFTREE before and after splitting is shown in Figure 5 And Figure 6 .

[0144] The data indexing method provided by the application can dynamically add indexes of new data to meet various production requirements.

[0145] Further, in one embodiment, updating the index table further includes:

[0146] Obtaining an index value of the to-be-deleted data in the original data set;

[0147] According to the index value of the to-be-deleted data, determining a second target leaf node to which each sub-vector of the to-be-deleted data belongs in the original data set of each clustering feature tree;

[0148] Deleting the relationship between the to-be-deleted data and the second target leaf node in each clustering feature tree, and performing node contraction judgment;

[0149] If the number of nodes after contraction is greater than a second preset value, a target node with the smallest distance from the second target leaf node is determined;

[0150] If the sum of the number of clustering feature nodes contained in the target node and the number of clustering feature nodes contained in the second target leaf node is less than a preset maximum clustering number, the index value of the target node is updated according to the index value of the target node and the index value of the second target leaf node;

[0151] Determine the second related data in the original data set affected, and update the second index value of the second related data based on the preset index construction method.

[0152] Optionally, if a certain data dx in the original data set X is to be deleted, the following steps are adopted to delete the index of the data in the original data set X:

[0153] Step C0, the dimension of the to-be-deleted index data dx is Q, which is divided into M sub-vectors xv m ,m=1,2,…,M;

[0154] Step C1, obtaining the index value nbi of dx:

[0155]

[0156] Step C2, according to the index value nbi, determining the dx in each CFTREEm The parent node of the leaf node is the second target leaf node, that is, if then

[0157] Step C3, let m'=m+1, assuming that the initial value of m is set to 1, in the CFTREE m , delete the relationship between dx and , and perform a node contraction judgment operation:

[0158] Step C4, if the number of nodes CFN after contraction is greater than the second preset value 1, that is, CFN>1, then calculate the inter-cluster distance D2 between and its sibling nodes, determine the sibling node with the minimum distance as the target node , and the CF node number CN pointed to by the target node k , if the cluster feature node number CN contained in the target node k and the cluster feature node number CN contained in the second target leaf node are less than the preset maximum clustering number B, that is, (CN k +CN)<B, then update the index value of the target node according to the index value of the target node and the index value of the second target leaf node, specifically, let

[0159] Step C5, repeat steps C3 to C4 until the M CFTREE index update operations are completed.

[0160] Step C6, determine the second related data in the affected original data set, and reconstruct the index value of the second related data based on the preset index construction method to update the second index value of the second related data.

[0161] The data index method provided by the application provides dynamic modification capability of index deletion, and can ensure data distribution and ensure the accuracy of the index when deleting the index.

[0162] The data index system provided by the application is described below, and the data index system described below can be correspondingly referred to the data index method described above.

[0163] Figure 7 is a structural diagram of the data index system provided by the application, as shown in Figure 7 , comprising: a data index module 710 and a data matching module 711;

[0164] The data index module 710 is used for determining the first index value of the to-be-matched data in the sample set according to the preset index construction method.

[0165] The data matching module 711 is configured to determine the first preset number of target data most matched with the to-be-matched data according to the index table of the original data set and the first index value in the sample set.

[0166] The index table is determined according to a second index value of the original data set.

[0167] The second index value is determined according to an index value of the original data set obtained according to a preset index construction method.

[0168] The data indexing module 710 includes an index construction submodule configured to divide the sample set into a second preset number of sub-vectors, and perform clustering on each sub-vector based on a BIRCH algorithm to obtain a clustering feature tree of each sub-vector.

[0169] The index value of each sub-vector is determined according to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree.

[0170] The index value of the sample set is determined according to the index value of each sub-vector.

[0171] The data indexing system provided by the application combines the vector quantization method and the BIRCH algorithm idea, divides the sample set into a plurality of sub-vectors first, uses the BIRCH algorithm to establish a clustering feature tree for each group of sub-vectors, and records the index value of each leaf node, and based on the index value of the to-be-matched data, the target data most matched with the to-be-matched data can be quickly retrieved from the index table of the original data set, and the data matching speed is improved.

[0172] Further, in an embodiment, the index construction submodule can be specifically used for:

[0173] The clustering feature tree is traversed, and the index point value of each node of all levels in the clustering feature tree is determined according to the depth and the maximum number of nodes.

[0174] The sample set is traversed, and the index value of the most adjacent node of each sub-vector in the current level is determined according to the index point value, and the index value of each sub-vector is determined by traversing each level.

[0175] The data indexing system provided by the application combines the vector quantization method and the BIRCH algorithm idea, and the high-dimensional large data set index construction method divides the data set into a plurality of sub-vectors in the index establishment process, uses the BIRCH algorithm to establish a CFTREE for each group of sub-vectors, and records the sample data pointed to by each leaf node, and realizes the index table construction of high-dimensional data.

[0176] Further, in an embodiment, the data matching module 711 can be further used for:

[0177] If the first index value of the sub-vector of the to-be-matched data and the second index value of the sub-vector of the original data set are the same at the same level, it is determined that the sub-vector of the to-be-matched data and the sub-vector of the original data set match;

[0178] According to the sub-vector of the original data set matched with the to-be-matched data, the matched data is determined.

[0179] If the number of the matched data is greater than or equal to the preset number, according to the similarity between the to-be-matched data and the matched data, the first preset number of target data most matched with the to-be-matched data are determined.

[0180] The data index system provided by the application can determine the data of the CF node to which the M-dimensional sub-vector of the to-be-matched data belongs as similar data when index retrieval is performed, and can automatically obtain the parent node information in the tree node according to the index vector when the amount of similar data is insufficient, thereby expanding the search range.

[0181] Further, in an embodiment, the system can further specifically include: a first index updating module, configured to determine the index value of the to-be-added data in the sample set according to a preset index construction method;

[0182] According to the index value of the to-be-added data, each sub-vector of the to-be-added data is determined to belong to a first target leaf node of a clustering feature tree of the original data set;

[0183] If the difference between the number of clustering feature nodes contained in the first target leaf node and the preset maximum clustering number is greater than a first preset value, the first target leaf node and a target parent node are split based on the preset index construction method;

[0184] The index value of the split target sub-node is added to the index table;

[0185] The first related data in the original data set affected is determined, and the second index value of the first related data is updated based on the preset index construction method;

[0186] The target parent node is determined according to the parent node of the first target leaf node.

[0187] The data index system provided by the application can realize the dynamic modification ability of adding data index, and can guarantee data distribution and ensure the accuracy of the index when the index is modified.

[0188] Further, in an embodiment, the second index updating module can be further configured to:

[0189] If the number of the split sub-nodes is less than the preset maximum sub-node threshold, the split target sub-node is directly split, and the index value of the split target sub-node is added to the index table;

[0190] If the number of split child nodes is greater than the preset maximum child node threshold, the target child node is split.

[0191] According to the depth of the cluster feature tree of the split original data set and the number of nodes of each layer of the cluster feature tree of the split original data set, the index value of the split target child node is determined, and the index value of the split target child node is added to the index table.

[0192] The data index system provided by the application can dynamically add indexes of new data to meet various production requirements.

[0193] Further, in one embodiment, the system can further include a second index updating module configured to obtain an index value of to-be-deleted data in the original data set.

[0194] According to the index value of the to-be-deleted data, each sub-vector of the to-be-deleted data is determined to belong to a second target leaf node of each cluster feature tree of the original data set.

[0195] In each cluster feature tree, the relationship between the to-be-deleted data and the second target leaf node is deleted, and node contraction judgment is performed.

[0196] If the number of contracted nodes is greater than the second preset value, a target node with the smallest distance from the second target leaf node is determined.

[0197] If the sum of the number of cluster feature nodes contained in the target node and the number of cluster feature nodes contained in the second target leaf node is less than the preset maximum cluster number, the index value of the target node is updated according to the index value of the target node and the index value of the second target leaf node.

[0198] Second related data in the original data set affected is determined, and the second index value of the second related data is updated based on a preset index construction method.

[0199] The data index system provided by the application provides dynamic modification capability for index deletion, and can guarantee data distribution and ensure the accuracy of the index when deleting the index.

[0200] Figure 8 is a schematic diagram of an entity structure of an electronic device provided by the application, as shown in Figure 8 The electronic device can include a processor 810, a communication interface 811, a memory 812 and a bus 813, wherein the processor 810, the communication interface 811 and the memory 812 can communicate with each other through the bus 813. The processor 810 can call the logical instructions in the memory 812 to execute the following method:

[0201] According to the preset index construction method, a first index value of the to-be-matched data in the sample set is determined;

[0202] According to the index table of the original data set in the sample set and the first index value, a first preset number of target data most matched with the to-be-matched data are determined;

[0203] The index table is determined according to a second index value of the original data set;

[0204] The second index value is determined according to the index value of the original data set obtained according to the preset index construction method;

[0205] The preset index construction method comprises:

[0206] The sample set is divided into a second preset number of sub-vectors, and each sub-vector is clustered based on the BRICH algorithm to obtain a clustering feature tree of each sub-vector;

[0207] According to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree, an index value of each sub-vector is determined;

[0208] According to the index value of each sub-vector, an index value of the sample set is determined.

[0209] In addition, the logical instructions in the memory described above can be implemented in the form of a software functional unit and sold or used as a stand-alone product, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer power supply screen (which can be a personal computer, a server, or a network power supply screen) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0210] Further, the present application discloses a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer can execute the data indexing method provided by the above-mentioned method embodiments, for example, comprising:

[0211] According to the preset index construction method, a first index value of the to-be-matched data in the sample set is determined;

[0212] determine, according to the index table and the first index value of the original data set in the sample set, a first preset number of target data that are most matched with the data to be matched;

[0213] The index table is determined according to a second index value of the original data set.

[0214] The second index value is determined according to an index value of the original data set obtained according to a preset index construction method.

[0215] The preset index construction method comprises:

[0216] divide the sample set into a second preset number of sub-vectors, and perform clustering on each sub-vector based on a BRICH algorithm to obtain a clustering feature tree of each sub-vector;

[0217] determine an index value of each sub-vector according to a depth of the clustering feature tree and a maximum number of nodes of each layer of the clustering feature tree;

[0218] determine an index value of the sample set according to the index value of each sub-vector.

[0219] On the other hand, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data indexing method provided by each of the above embodiments, for example comprising:

[0220] determine, according to a preset index construction method, a first index value of the data to be matched in the sample set;

[0221] determine, according to the index table and the first index value of the original data set in the sample set, a first preset number of target data that are most matched with the data to be matched;

[0222] The index table is determined according to a second index value of the original data set.

[0223] The second index value is determined according to an index value of the original data set obtained according to a preset index construction method.

[0224] The preset index construction method comprises:

[0225] divide the sample set into a second preset number of sub-vectors, and perform clustering on each sub-vector based on a BRICH algorithm to obtain a clustering feature tree of each sub-vector;

[0226] determine an index value of each sub-vector according to a depth of the clustering feature tree and a maximum number of nodes of each layer of the clustering feature tree;

[0227] determine an index value of the sample set according to the index value of each sub-vector.

[0228] The system embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to a predetermined number of network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0229] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer power supply screen (which can be a personal computer, a server, or a network power supply screen, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0230] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data indexing method, characterized by, The method comprises the following steps: According to the preset index construction method, the first index value of the to-be-matched data in the sample set is determined; According to the index table of the original data set in the sample set and the first index value, the first preset number of target data most matched with the to-be-matched data are determined; The index table is determined according to the second index value of the original data set; The second index value is determined according to the index value of the original data set obtained according to the preset index construction method; The preset index construction method comprises the following steps: The sample set is divided into a second preset number of sub-vectors, and each sub-vector is clustered based on the BRICH algorithm to obtain the clustering feature tree of each sub-vector; According to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree, the index value of each sub-vector is determined; According to the index value of each sub-vector, the index value of the sample set is determined; According to the index table of the original data set in the sample set and the first index value, the first preset number of target data most matched with the to-be-matched data are determined, which comprises the following steps: If the first index value of the sub-vector of the to-be-matched data at the same level is the same as the second index value of the sub-vector of the original data set, it is determined that the sub-vector of the to-be-matched data matches the sub-vector of the original data set; According to the sub-vector of the original data set matched with the to-be-matched data, the matching data is determined; If the number of matching data is greater than or equal to the preset number, the first preset number of target data most matched with the to-be-matched data are determined according to the similarity between the to-be-matched data and the matching data.

2. The data indexing method of claim 1, wherein, According to the depth of the clustering feature tree and the maximum number of nodes of each layer of the clustering feature tree, the index value of each sub-vector is determined, which comprises the following steps: Traverse the clustering feature tree, and determine the index point value of each node at all levels in the clustering feature tree according to the depth and the maximum number of nodes; Traverse the sample set, and determine the index value of each sub-vector and the nearest node at the current level according to the index point value. Traverse each level to determine the index value of each sub-vector.

3. The data indexing method of claim 1, wherein, Updating the index table comprises the following steps: According to the preset index construction method, the index value of the to-be-added data in the sample set is determined; According to the index value of the to-be-added data, the first target leaf node to which each sub-vector of the to-be-added data belongs in each clustering feature tree of the original data set is determined; If the difference between the number of clustering feature nodes contained in the first target leaf node and the preset maximum clustering number is greater than a first preset value, the first target leaf node and the target parent node are split based on the preset index construction method; The index value of the split target sub-node is added to the index table; The first related data in the original data set affected is determined, and the second index value of the first related data is updated based on the preset index construction method; The target parent node is determined according to the parent node of the first target leaf node.

4. The data indexing method of claim 3, wherein, The index value of the split target sub-node is added to the index table, which comprises the following steps: If the number of the split sub-nodes is less than the preset maximum sub-node threshold, directly split the target sub-node, and add the index value of the split target sub-node to the index table; If the number of the split sub-nodes is greater than the preset maximum sub-node threshold, split the target sub-node; According to the depth of the cluster feature tree of the split original data set and the number of nodes of each layer of the cluster feature tree of the split original data set, determine the index value of the split target sub-node, and add the index value of the split target sub-node to the index table.

5. The data indexing method of claim 1, wherein, Updating the index table further comprises: Obtaining the index value of the data to be deleted in the original data set; According to the index value of the data to be deleted, determining the second target leaf node to which each sub-vector of the data to be deleted belongs in each cluster feature tree of the original data set; Deleting the relationship between the data to be deleted and the second target leaf node in each cluster feature tree, and performing node contraction judgment; If the number of contracted nodes is greater than a second preset value, determine the target node with the minimum distance from the second target leaf node; If the sum of the number of cluster feature nodes contained in the target node and the number of cluster feature nodes contained in the second target leaf node is less than a preset maximum clustering number, update the index value of the target node according to the index value of the target node and the index value of the second target leaf node; Determine the second related data in the original data set affected, and update the second index value of the second related data based on the preset index construction method.

6. The data indexing method according to any one of claims 1-5, wherein, The index value comprises: Index identification and the number of bits of the index identification.

7. A data indexing system characterized by, Comprise: Data indexing module and data matching module; The data indexing module is configured to determine a first index value of data to be matched in a sample set according to a preset index construction method; The data matching module is configured to determine a first preset number of target data most matched with the data to be matched according to an index table of an original data set in the sample set and the first index value; The index table is determined according to a second index value of the original data set; The second index value is determined according to the index value of the original data set obtained according to the preset index construction method; The data indexing module comprises an index construction submodule configured to divide the sample set into a second preset number of sub-vectors, and cluster each sub-vector based on a BRICH algorithm to obtain a cluster feature tree of each sub-vector; According to the depth of the cluster feature tree and the maximum number of nodes of each layer of the cluster feature tree, determine the index value of each sub-vector; According to the index value of each sub-vector, determine the index value of the sample set; The data matching module is further configured to: if the first index value of the sub-vector of the to-be-matched data and the second index value of the sub-vector of the original data set are the same, determine that the sub-vector of the to-be-matched data and the sub-vector of the original data set match; determine matching data according to the sub-vector of the original data set that matches the to-be-matched data; and if the number of the matching data is greater than or equal to the preset number, determine the first preset number of target data that best match the to-be-matched data according to the similarity between the to-be-matched data and the matching data.

8. An electronic device comprising a processor and a memory having a computer program stored therein, characterized in that, The processor executes the computer program to implement the steps of the data indexing method in any one of claims 1 to 6.

9. A processor-readable storage medium, comprising: The processor readable storage medium stores a computer program configured to cause the processor to execute the steps of the data indexing method in any one of claims 1 to 6.

Images