Improved algorithm for rapid similarity search of high-dimensional data

Abstract

The invention provides an improved algorithm for rapid similarity search of high-dimensional data. The improved algorithm comprises the steps of data preprocessing, improved sparse coding, construction of a hierarchical index structure, rapid similarity search and result sorting and returning. According to the method, standardization and principal component analysis dimension reduction are performed on the data, so that the dimension and calculation complexity of the data are reduced; the improved ball tree index structure is utilized to finely divide a high-density region, and a pruning strategy is adopted in the search process, so that the query efficiency and accuracy are improved. The method is suitable for an application scene in which rapid similarity search needs to be carried out in the high-dimensional data.

Description

Technical Field

[0001] The present invention relates to the field of computer software algorithms, and in particular to an improved algorithm for fast similarity search of high-dimensional data. Background Art

[0002] With the development of information technology, massive high-dimensional data are widely present in various fields, such as image processing, text analysis, bioinformatics, etc. In high-dimensional space, data processing and analysis face huge challenges, mainly reflected in the sharp increase in computational complexity and storage requirements, a phenomenon known as the "curse of dimensionality."

[0003] Traditional similarity search algorithms, such as linear scan, are extremely inefficient on high-dimensional data sets. To solve this problem, researchers have proposed a variety of high-dimensional index structures and approximate search algorithms, such as Locality-Sensitive Hashing (LSH) and kd-tree. However, these methods still have problems such as low query accuracy and low computational efficiency in high-dimensional situations.

[0004] Therefore, there is an urgent need for an improved algorithm that can efficiently and accurately perform similarity search in high-dimensional data. Summary of the invention

[0005] The present invention aims to overcome the deficiencies of the prior art and provide an improved algorithm for fast similarity search of high-dimensional data, which improves search efficiency, reduces computational complexity, and maintains high search accuracy by introducing improved sparse coding and hierarchical index structure.

[0006] To achieve the above object, the present invention proposes the following technical solutions:

[0007] An improved algorithm for fast similarity search of high-dimensional data includes the following steps:

[0008] Step 1, data preprocessing: normalize or standardize the original high-dimensional data to eliminate the scale differences between different dimensions and improve the comparability of the data.

[0009] Step 2, improved sparse coding: Using the improved sparse coding algorithm, high-dimensional data is represented as low-dimensional sparse vectors, reducing the data dimension while retaining the main features of the original data.

[0010] Step 3: Construction of hierarchical index structure: Based on the sparse coding results, a hierarchical index structure is constructed, including a coarse-grained index layer and a fine-grained index layer, to speed up the search process.

[0011] Step 4, fast similarity search: When querying, use the hierarchical index structure to first quickly locate potential similar data at the coarse-grained layer, and then accurately calculate the similarity at the fine-grained layer to find the most similar data points.

[0012] Step 5: sorting and returning results: sort the search results according to the similarity scores and return them to the user.

[0013] The present invention has the following beneficial effects:

[0014] (1) Improve search efficiency: Sparse coding can reduce data dimensions, and the hierarchical index structure can accelerate the search process, thereby improving search efficiency.

[0015] (2) Improve search accuracy: Sparse coding retains the main features of the original data and can ensure the accuracy of search results.

[0016] (3) Strong scalability: The algorithm provided by the present invention is suitable for large-scale, high-dimensional data sets and has good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an overall flow chart of the improved algorithm for fast similarity search of high-dimensional data described in the present invention.

[0018] Figure 2 Flowchart of the improved sparse coding algorithm described in the present invention.

[0019] Figure 3 It is a schematic diagram of the hierarchical index structure of the present invention.

[0020] Figure 4 It is a schematic diagram of the similarity search process of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] 1. Embodiments of the technical solution of the present invention

[0023] like Figure 1-Figure 4 As shown, the improved algorithm for fast similarity search of high-dimensional data according to the present invention comprises the following steps:

[0024] Step 1: Data Preprocessing

[0025] For a high-dimensional dataset D = {x1, x2, ..., x n}, where x i ∈R d , normalization is performed so that the data has the same scale in each dimension.

[0026] Commonly used normalization methods:

[0027] (1) Minimum-maximum normalization:

[0028] (2) Standardization (Z-score):

[0029] Where n represents the number of data points in the dataset; x i represents the data points in the data set; R represents the set of real numbers; d is the dimension of the data space; R d represents a vector space with d real coordinates; represents the normalized data point; x min 、x max They represent the minimum and maximum values ​​in each dimension respectively; μ and σ represent the mean vector and standard deviation vector respectively.

[0030] Step 2: Improved sparse coding

[0031] Sparse coding aims to find an overcomplete basis Φ∈R d×m (m>d), so that each data point can be approximately represented by a linear combination of a few basis vectors. Where m represents the number of columns of the overcomplete basis, d represents the number of rows, and R d×m represents a matrix with d rows and m columns, and R represents the set of real numbers.

[0032] Traditional sparse coding solution:

[0033] Among them, s i represents the coefficient vector of sparse coding. λ represents the sparse regularization parameter, which controls the degree of sparsity; Φ represents the overcomplete basis matrix, which is obtained by training the dictionary learning algorithm.

[0034] The present invention adopts an improved sparse coding algorithm and introduces an accelerated iterative convergence method, namely, a fast iterative soft threshold algorithm (FISTA).

[0035] FISTA algorithm steps:

[0036] (1) Initialization: Setting the initial value in, represents the initial sparse coding; t0 represents an initial parameter of the iterative algorithm, which is usually initialized to 1 in the FISTA algorithm to accelerate the iterative convergence; y0 represents the initial value of the iteration, which is used to calculate the subsequent yk value and is equal to

[0037] (2) Iterative update: for k = 0, 1, 2, ..., k max , until convergence, perform the following updates:

[0038]

[0039]

[0040]

[0041] Where k represents the number of similar data points returned; k max Indicates the maximum number of iterations, which is used to control the maximum number of iterations of the algorithm to avoid infinite loops. represents the sparse coding at the kth iteration; represents the updated sparse code; yk represents the auxiliary variable at the kth iteration, which is used to accelerate the update of the sparse code; L represents the Lipschitz constant of the gradient, usually L=|Φ T The largest eigenvalue of Φ|2, T is the transpose sign, Φ r represents the transposed matrix of matrix Φ; t k and t k+1 They represent the auxiliary parameters at the kth and k+1th iterations respectively. The FISTA algorithm updates t k To accelerate convergence; y k+1 Represents the auxiliary variable at the k+1th iteration, used to calculate the next sparse code

[0042] Shrink is a soft threshold function, defined as:

[0043] Shrink(x, θ)=sign(x)·max(|x|-θ, 0), where θ represents the threshold value of the Shrink function.

[0044] (3) Convergence condition judgment: After each iteration, check whether one of the following convergence conditions is met. If so, stop the iteration:

[0045] The change of the objective function: Where ∈ is a small threshold, usually 10 -5 or smaller to ensure the convergence of the objective function.

[0046] The amount of change in sparse coding: Where δ is the threshold of control accuracy, usually 10 -5 .

[0047] Maximum number of iterations k max :The number of iterations k reaches the set maximum number of iterations k max .

[0048] (4) Output sparse coding results.

[0049] Step 3: Construction of hierarchical index structure

[0050] (1) Based on the sparse coding results {s i}, build a hierarchical index structure.

[0051] (2) Coarse-grained indexing layer:

[0052] Hash function: The non-zero element position of the sparse coding vector is used as the input of the hash function to generate a hash code: hash function h(s i )=Index of Non-zero Elements(s i )

[0053] Bucket: Data points with the same hash code are grouped together to form a bucket, which is determined by the sparsity and dimension of the sparse coding.

[0054] (3) Fine-grained indexing layer:

[0055] Within each bucket, further subdivision is performed using a space partitioning algorithm (such as K-means clustering) based on the value of the sparse coding coefficient.

[0056] Step 4: Fast Similarity Search

[0057] For a query data point q, the following steps are performed:

[0058] (1) Preprocessing and sparse coding: Normalize q to obtain Using the same sparse coding algorithm, calculate the sparse coding s of the query data q q , used in the subsequent similarity search process.

[0059] (2) Coarse-grained search: Calculate the sparse code s q The corresponding hash bucket h(s q ), locate the corresponding bucket. h(s q ) function according to s q The non-zero element positions of are used to map the query data to the corresponding buckets for coarse-grained search.

[0060] (3) Fine-grained search: In a bucket, select a set of candidate data points based on a pre-built fine-grained index. With candidate data points The similarity.

[0061] Similarity measure:

[0062] Cosine Similarity: in Represents the normalized query data The transposed vector of . In the calculation formula of cosine similarity, With candidate data points Perform dot product to calculate the similarity between the two.

[0063] Euclidean distance:

[0064] (4) Result sorting and return:

[0065] Sort by similarity score in descending order, select the first k data points as the final result, and return the sorted similar data points and their similarity scores to the user.

[0066] 2. Application Examples of the Invention

[0067] In order to verify the effectiveness of the algorithm of the present invention, the inventors conducted the following experimental tests.

[0068] Experimental settings: The dataset uses a public high-dimensional dataset, such as the MNIST handwritten digit image dataset (28x28 pixels, 784 dimensions). Comparison algorithm: The algorithm of the present invention is compared with the traditional LSH algorithm and KD tree algorithm. Evaluation indicators: search time, search precision (accuracy), memory consumption.

[0069] Experimental results: (1) Search time: The search time of the algorithm of the present invention on high-dimensional data is significantly lower than that of the LSH algorithm and the KD tree algorithm. (2) Search accuracy: The search accuracy of the algorithm of the present invention is comparable to that of the KD tree algorithm and is better than that of the LSH algorithm. (3) Memory consumption: Due to the sparsity of sparse coding, the index structure of the present invention occupies less memory.

[0070] Parameter optimization: (1) Sparse regularization parameter λ: Select an appropriate λ value through cross-validation to strike a balance between sparsity and representation accuracy. (2) Overcomplete basis Φ: Trained through a dictionary learning algorithm to effectively represent data.

Claims

1. An improved algorithm for fast similarity search of high-dimensional data, characterized in that: The following steps are involved: Step 1, data preprocessing: normalize or standardize the original high-dimensional data to eliminate the scale differences between different dimensions and improve the comparability of the data; Step 2: Improved sparse coding: Using the improved sparse coding algorithm, high-dimensional data is represented as low-dimensional sparse vectors to reduce the data dimension while retaining the main features of the original data; Step 3: Construction of hierarchical index structure: Based on the sparse coding results, a hierarchical index structure is constructed, including a coarse-grained index layer and a fine-grained index layer, to accelerate the search process; Step 4, fast similarity search: When querying, use the hierarchical index structure to first quickly locate potential similar data at the coarse-grained layer, then accurately calculate the similarity at the fine-grained layer to find the most similar data points; Step 5: sorting and returning results: sort the search results according to the similarity scores and return them to the user.

2. The improved algorithm for fast similarity search of high-dimensional data according to claim 1, characterized in that: The step 2 specifically includes the following steps: (1) Initialization: Setting the initial value in, represents the initial sparse coding, t0 represents an initial parameter of the iterative algorithm, and y0 represents the initial value of the iteration; (2) Iterative update: for k = 0, 1, 2, ..., k max , until convergence, perform the following updates: Update sparse coding Update auxiliary parameters Update auxiliary variables Where k represents the number of similar data points returned; k max Indicates the maximum number of iterations; represents the sparse coding at the kth iteration; y k represents the auxiliary variable at the kth iteration; L represents the Lipschitz constant of the gradient, usually L=|Φ T The largest eigenvalue of Φ|2; T is the transpose symbol, Φ T represents the transposed matrix of the overcomplete basis matrix Φ; t k represents the auxiliary parameters at the kth time; (3) Convergence condition judgment: After each iteration, check whether one of the following convergence conditions is met. If so, stop the iteration: The change of the objective function: Where ∈ is a small threshold that controls the change of the objective function; The amount of change in sparse coding: Where δ is the threshold of control accuracy; Maximum number of iterations k max :The number of iterations k reaches the set maximum number of iterations k max ; (4) Output sparse coding result {s i }.

3. The improved algorithm for fast similarity search of high-dimensional data according to claim 2, characterized in that: The Shrink is a soft threshold function, defined as: Shrink k(x,θ)=sig n(x)·max(|x|-θ,0), where θ represents the threshold of the Shrink function.

4. The improved algorithm for fast similarity search of high-dimensional data according to claim 1, characterized in that: The step 3 specifically comprises the following steps: (1) Based on the sparse coding results {s i }, build a hierarchical index structure; (2) Coarse-grained indexing layer: The non-zero element positions of the sparse coding vector are used as the input of the hash function to generate a hash code: the hash function h(s i )=Index of Non-zero Elements(s i ); data points with the same hash code are grouped into one category to form buckets; (3) Fine-grained indexing layer: Within each bucket, a spatial partitioning algorithm is used to further subdivide the bucket based on the value of the sparse coding coefficient.

5. The improved algorithm for fast similarity search of high-dimensional data according to claim 1, characterized in that: The specific method of step 4 is as follows: (1) Preprocessing and sparse coding: Normalize the query data point q to obtain Using the same sparse coding algorithm, calculate the sparse coding s of the query data q q ; (2) Coarse-grained search: Calculate the sparse code s q The corresponding hash bucket h(s q ), locate the corresponding bucket; (3) Fine-grained search: In a bucket, select a set of candidate data points based on a pre-built fine-grained index; calculate With candidate data points similarity; (4) Result sorting: Arrange in descending order of similarity scores and select the first k data points as the final result.

6. The improved algorithm for fast similarity search of high-dimensional data according to claim 5, characterized in that: The similarity is calculated using the cosine similarity algorithm: in Represents the normalized query data The transposed vector of .

7. The improved algorithm for fast similarity search of high-dimensional data according to claim 5, characterized in that: The similarity is calculated using the Euclidean distance algorithm:

Images