Archive retrieval method and system

Abstract

The invention relates to the field of archive management, in particular to an archive retrieval method and system, and the method comprises the steps: carrying out the feature extraction of an archive, selecting a plurality of subsets from the archive, constructing a similarity matrix of the subsets through the extracted features, calculating the Laplacian matrix of the subsets according to the similarity matrix of the subsets, and obtaining the feature value of the Laplacian matrix; obtaining the number of clusters based on the feature values of all the subsets, obtaining anchor points according to the number of clusters and the feature values of the subsets, and clustering the archives by using the number of clusters and the anchor points; and a retrieval keyword input by the user is converted into a feature vector, the distance between the feature vector and the center vector of each cluster is calculated, and the file in the cluster with the minimum distance is returned to the user.

Description

Technical Field

[0001] The present invention relates to the field of retrieval, and in particular to an archive retrieval method and system. Background Art

[0002] Traditional archive retrieval methods mainly rely on keyword matching and Boolean retrieval, which have problems such as insufficient semantic understanding and incomplete retrieval results. The emergence of massive archive data makes it difficult for traditional retrieval methods to meet the growing needs of users. Based on the idea of ​​graph theory, the spectral clustering algorithm regards archive data as nodes in the graph, constructs a similarity matrix by calculating the similarity between nodes, and uses the eigenvectors of the matrix for clustering, thereby realizing automatic classification and retrieval of archives. Compared with traditional clustering algorithms, such as the K-means algorithm, spectral clustering has stronger adaptability to data distribution, can effectively process non-convex data sets and high-dimensional data, and is not easy to fall into local optimal solutions. However, spectral clustering has high computational complexity and is sensitive to parameter selection, which limits its further promotion in practical applications. Improving the spectral clustering algorithm to improve its efficiency and robustness is one of the important research directions in the field of archive retrieval. Summary of the invention

[0003] In view of the problem that the current spectral clustering has high computational complexity in archive retrieval, in a first aspect of the present invention, a method for archive retrieval is provided, the method comprising: Extract features from the archives, select multiple subsets from the archives, construct a similarity matrix of the subsets using the extracted features, and calculate the Laplace matrix of the subsets based on the similarity matrix of the subsets to obtain the eigenvalues ​​of the Laplace matrix; Obtaining the number of clusters based on the feature values ​​of all subsets, obtaining anchor points according to the number of clusters and the feature values ​​of the subsets, and clustering the archives using the number of clusters and the anchor points; The search keywords input by the user are converted into feature vectors, the distance between the feature vectors and the center vector of each cluster is calculated, and the archives in the cluster with the smallest distance are returned to the user.

[0004] Optionally, the number of clusters obtained based on the feature values ​​of all subsets is specifically: Select the subset with the minimum number of eigenvalues ​​among all subsets, and for the remaining subsets, retain the eigenvalues ​​with the minimum number of eigenvalues ​​mentioned above for each subset; For each subset, the difference between adjacent eigenvalues ​​is calculated to obtain a difference sequence, and the average of the difference sequences of all subsets is calculated to obtain an average sequence; The position of the maximum value in the average value sequence is found from the average value sequence, and the sequence number of the position is used as the number of clusters.

[0005] Optionally, obtaining the anchor point according to the number of clusters and the feature value of the subset is specifically: Calculate the sum of the eigenvalues ​​of the number of clusters in front of each subset, and calculate the union of multiple subsets with the smallest sum of eigenvalues; The difference between the union of multiple subsets with the smallest sum of eigenvalues ​​and the intersection of the remaining subsets is taken as the anchor point set, and each element in the anchor point set is an anchor point.

[0006] Optionally, the distance between the feature vector and the center vector of each cluster is calculated as follows: Calculate the distance from the eigenvector to the center vector of each cluster and obtain the eigenvalues ​​of the Laplacian matrix of all anchor points; The product of the distance and the normalized eigenvalue is calculated, and the result of subtracting the product from the distance is taken as the distance between the eigenvector and the center vector of the cluster.

[0007] Optionally, returning the archive in the cluster with the smallest distance to the user is specifically: The distance between the feature vector of the archive in the cluster with the smallest distance and the center vector of the cluster is calculated, the archive in the cluster with the smallest distance is sorted in ascending order of the distance, and the archive is returned to the user according to the sorting result.

[0008] In a second aspect of the present invention, there is provided an archive retrieval system, the system comprising: The feature extraction module is used to extract features from the archives, select multiple subsets from the archives, construct a similarity matrix of the subsets using the extracted features, and calculate the Laplace matrix of the subsets based on the similarity matrix of the subsets to obtain the eigenvalues ​​of the Laplace matrix; A clustering module, used to obtain the number of clusters based on the feature values ​​of all subsets, obtain anchor points according to the number of clusters and the feature values ​​of the subsets, and cluster the archives using the number of clusters and the anchor points; The search result return module is used to convert the search keywords input by the user into feature vectors, calculate the distance between the feature vectors and the center vector of each cluster, and return the archives in the cluster with the smallest distance to the user.

[0009] Optionally, the number of clusters obtained based on the feature values ​​of all subsets is specifically: Select the subset with the minimum number of eigenvalues ​​among all subsets, and for the remaining subsets, retain the eigenvalues ​​with the minimum number of eigenvalues ​​mentioned above for each subset; For each subset, the difference between adjacent eigenvalues ​​is calculated to obtain a difference sequence, and the average of the difference sequences of all subsets is calculated to obtain an average sequence; The position of the maximum value in the average value sequence is found from the average value sequence, and the sequence number of the position is used as the number of clusters.

[0010] Optionally, obtaining the anchor point according to the number of clusters and the feature value of the subset is specifically: Calculate the sum of the eigenvalues ​​of the number of clusters in front of each subset, and calculate the union of multiple subsets with the smallest sum of eigenvalues; The difference between the union of multiple subsets with the smallest sum of eigenvalues ​​and the intersection of the remaining subsets is taken as the anchor point set, and each element in the anchor point set is an anchor point.

[0011] Optionally, the distance between the feature vector and the center vector of each cluster is calculated as follows: Calculate the distance from the eigenvector to the center vector of each cluster and obtain the eigenvalues ​​of the Laplacian matrix of all anchor points; The product of the distance and the normalized eigenvalue is calculated, and the result of subtracting the product from the distance is taken as the distance between the eigenvector and the center vector of the cluster.

[0012] Optionally, returning the archive in the cluster with the smallest distance to the user is specifically: The distance between the feature vector of the archive in the cluster with the smallest distance and the center vector of the cluster is calculated, the archive in the cluster with the smallest distance is sorted in ascending order of the distance, and the archive is returned to the user according to the sorting result.

[0013] Compared with the prior art, the present invention has the following advantages: 1. By combining the eigenvalues ​​of the Laplacian matrix of multiple subsets, the number of clusters can be estimated more robustly, reducing the impact of noise or bias from a single subset; 2. By selecting the smallest sum of eigenvalues ​​and calculating the difference set to obtain the anchor point set, more representative data points are used as anchor points to prevent the problem of noise as anchor points leading to subsequent clustering errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of a process flow of an archive retrieval method provided in an embodiment of the present application; Figure 2 A schematic diagram of a process for determining the number of clusters provided in an embodiment of the present application; Figure 3 A schematic diagram of an archive search page provided in an embodiment of the present application; Figure 4 A schematic diagram of the distance from the search term to the cluster center provided in the embodiment of the present application; Figure 5 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] The first embodiment provides a file retrieval method, such as Figure 1 As shown, the method includes: S1, extracting features from the archives, selecting multiple subsets from the archives, constructing a similarity matrix of the subsets using the extracted features, and calculating the Laplacian matrix of the subsets based on the similarity matrix of the subsets to obtain the eigenvalues ​​of the Laplacian matrix; Archives usually contain various raw data, such as text, numbers, images, etc., from which numerical representations that can represent the essential characteristics of the data, namely feature vectors, are extracted. For texts in archives, methods such as bag-of-words model, TF-IDF, word embedding, etc. are used to extract text features; for images in archives, methods such as SIFT, SURF, CNN, etc. are used to extract image features or identify objects in images, and objects are encoded to obtain features; for values ​​in archives, original values ​​are used or standardized, normalized, etc. are performed. In another embodiment, keywords of the archives are extracted, and the keywords are encoded to obtain the features of the archives. In one embodiment, the archives are text archives or image archives.

[0017] For large-scale data sets, it may be very time-consuming to perform calculations directly on the entire data set. By dividing the subsets, the computational complexity can be reduced. Methods for selecting subsets include but are not limited to random selection, selection based on time windows, etc. In one embodiment, the number of archives in each subset is different or the same. For each subset, the similarity between the archives is calculated based on the eigenvectors of the archives therein, and a similarity matrix is ​​constructed. Based on the similarity matrix, the Laplace matrix of the subset is calculated. Then the Laplace matrix is ​​eigen-decomposed to obtain its eigenvalues.

[0018] S2, obtaining the number of clusters based on the feature values ​​of all subsets, obtaining anchor points according to the number of clusters and the feature values ​​of the subsets, and clustering the archives using the number of clusters and the anchor points; Determining the appropriate number of clusters is the key to cluster analysis. If the number of clusters is not selected properly, the clustering results will be inaccurate. In one embodiment, for each subset, the gaps between the eigenvalues ​​of its Laplace matrix, that is, the differences between adjacent eigenvalues, are calculated. Generally, a larger gap means that the data may have an obvious clustering structure within the subset. The eigenvalue gap information of all subsets is summarized, such as calculating the average gap or finding the maximum value among all gaps, to obtain the appropriate number of clusters. For example, if there is a large gap between the second and third eigenvalues ​​of multiple subsets, the number of clusters is set to 2. It should be noted that when calculating the eigenvalue gap, the eigenvalues ​​are first sorted, and preferably, the eigenvalues ​​are sorted in order from small to large.

[0019] In another embodiment, the number of clusters is obtained based on the feature values ​​of all subsets, such as Figure 2 As shown, specifically: S21, selecting a subset with the minimum number of eigenvalues ​​from all subsets, and for the remaining subsets, retaining the eigenvalues ​​with the minimum number of eigenvalues ​​in each subset; S22, for each subset, calculating the difference between adjacent eigenvalues ​​to obtain a difference sequence, and calculating the average of the difference sequences of all subsets to obtain an average sequence; S23, finding the position of the maximum value in the average value sequence from the average value sequence, and taking the sequence number of the position as the number of clusters.

[0020] Different subsets may contain different numbers of data points, so the size of their Laplacian matrices may also be different, resulting in different numbers of eigenvalues. For subsequent unified comparison, the number of eigenvalues ​​of all subsets needs to be unified to a minimum value. For example, there are three subsets, and their numbers of eigenvalues ​​are 5, 7, and 4 respectively. The minimum number of eigenvalues ​​is 4. For subsets with eigenvalues ​​of 5 and 7, only the first 4 eigenvalues ​​are retained, and then all subsets have only 4 eigenvalues. It should be noted that the eigenvalues ​​of each subset will also be sorted here. Preferably, the eigenvalues ​​are sorted in order from small to large, that is, the eigenvalues ​​with the smallest first 4 eigenvalues ​​are selected.

[0021] The difference between the eigenvalues ​​reflects the clustering trend of the data points within the subset. For each subset, the difference between two adjacent eigenvalues ​​in its eigenvalue sequence is calculated to obtain a difference sequence. Then, the difference values ​​of the corresponding positions of the difference sequences of all subsets are averaged to obtain an average difference sequence. In the obtained average difference sequence, the largest difference is found, and its position in the sequence, i.e., the index, is recorded. The serial number of this position is the estimated number of clusters. It should be noted that the index or position of the average difference sequence starts from 1. The index of the first element of the average difference sequence is 1, and so on. If the maximum value is located at the fourth position of the average value, the number of clusters is 4.

[0022] After the number of clusters is determined, some representative data points are selected from the data set as anchor points. Preferably, the sum of the minimum eigenvalues ​​of the number of clusters shown before each subset is calculated, and the subset with the smallest sum is selected. The data points inside the smallest subset are more concentrated, more structured, and more suitable as anchor points.

[0023] In order to prevent the anchor point from being noise, in another embodiment, the anchor point is obtained according to the number of clusters and the feature value of the subset, specifically: Calculate the sum of the eigenvalues ​​of the number of clusters in front of each subset, and calculate the union of multiple subsets with the smallest sum of eigenvalues; The difference between the union of multiple subsets with the smallest sum of eigenvalues ​​and the intersection of the remaining subsets is taken as the anchor point set, and each element in the anchor point set is an anchor point.

[0024] For each subset, calculate the sum of the minimum eigenvalues ​​of the number of clusters mentioned above in its Laplacian matrix, select at least one subset with the smallest sum of eigenvalues, and merge all the data points contained in these selected subsets to form a union. Calculate the intersection of the obtained union and the remaining subsets, and then subtract this intersection from the union. The difference obtained is the final anchor point set. Remove those data points that belong to both the union of the subsets with the smallest sum of eigenvalues ​​and the remaining subsets. These data points may be located near the boundaries of different clusters and are less representative as anchor points.

[0025] After obtaining the number of clusters and anchor points, the spectral clustering algorithm based on anchor points is used to complete the clustering of archives.

[0026] S3, converts the search keywords input by the user into feature vectors, calculates the distance between the feature vectors and the center vector of each cluster, and returns the archives in the cluster with the smallest distance to the user.

[0027] The search keywords input by the user include but are not limited to words, phrases, or sentences. The keywords input by the user are converted into feature vectors by word embedding or word bag. After clustering the archives, each cluster has a center vector. The distance between the feature vector of the search keyword and the center vector of the cluster is calculated, and the cluster with the smallest distance to the user keyword feature vector is selected, and all archives in the cluster are returned to the user as the search result. In one embodiment, the present invention is applicable to fuzzy retrieval of archives, such as Figure 3 shown.

[0028] In an optional embodiment, the distance between the feature vector and the center vector of each cluster is calculated as follows: Calculate the distance from the eigenvector to the center vector of each cluster and obtain the eigenvalues ​​of the Laplacian matrix of all anchor points; The product of the distance and the normalized eigenvalue is calculated, and the result of subtracting the product from the distance is taken as the distance between the eigenvector and the center vector of the cluster.

[0029] Use distance measurement methods such as Euclidean distance, cosine distance, etc. to calculate the distance from the feature vector to each cluster center vector. Figure 4 The diagram shows the distance from the search term to the cluster center. After obtaining the anchor point, the eigenvalue of the Laplace matrix of the archive set formed by the anchor point is further calculated. The eigenvalue usually reflects the compactness of the cluster or the data distribution. The eigenvalue is normalized, and the product of the distance and the normalized eigenvalue is calculated. The result of the distance minus the product is taken as the distance between the eigenvector and the center vector of the cluster. For example, the distance from the eigenvector of the search term to the center vectors of clusters 1 and 2 is 5. The eigenvalue of cluster 1 is assumed to be 0.3, and the eigenvalue of cluster 2 is assumed to be 0.6. In general, cluster 1 is more compact and cluster 2 is looser. The result of the distance minus the product corresponding to cluster 1 is 3.5, and the result of the distance minus the product corresponding to cluster 2 is 2. The eigenvector of the search term is closer to cluster 2, and the archive returned to the user is the archive in cluster 2.

[0030] After obtaining the cluster with the shortest distance, the distance between the feature vector of the archive in the cluster with the shortest distance and the center vector of the cluster is calculated, and the archive in the cluster with the shortest distance is sorted in ascending order of the distance, and the archive is returned to the user according to the sorting result.

[0031] The second embodiment provides an archive retrieval system, the system comprising: The feature extraction module is used to extract features from the archives, select multiple subsets from the archives, construct a similarity matrix of the subsets using the extracted features, and calculate the Laplace matrix of the subsets based on the similarity matrix of the subsets to obtain the eigenvalues ​​of the Laplace matrix; A clustering module, used to obtain the number of clusters based on the feature values ​​of all subsets, obtain anchor points according to the number of clusters and the feature values ​​of the subsets, and cluster the archives using the number of clusters and the anchor points; The search result return module is used to convert the search keywords input by the user into feature vectors, calculate the distance between the feature vectors and the center vector of each cluster, and return the archives in the cluster with the smallest distance to the user.

[0032] Optionally, the number of clusters obtained based on the feature values ​​of all subsets is specifically: Select the subset with the minimum number of eigenvalues ​​among all subsets, and for the remaining subsets, retain the eigenvalues ​​with the minimum number of eigenvalues ​​mentioned above for each subset; For each subset, the difference between adjacent eigenvalues ​​is calculated to obtain a difference sequence, and the average of the difference sequences of all subsets is calculated to obtain an average sequence; The position of the maximum value in the average value sequence is found from the average value sequence, and the sequence number of the position is used as the number of clusters.

[0033] Optionally, obtaining the anchor point according to the number of clusters and the feature value of the subset is specifically: Calculate the sum of the eigenvalues ​​of the number of clusters in front of each subset, and calculate the union of multiple subsets with the smallest sum of eigenvalues; The difference between the union of multiple subsets with the smallest sum of eigenvalues ​​and the intersection of the remaining subsets is taken as the anchor point set, and each element in the anchor point set is an anchor point.

[0034] Optionally, the distance between the feature vector and the center vector of each cluster is calculated as follows: Calculate the distance from the eigenvector to the center vector of each cluster and obtain the eigenvalues ​​of the Laplacian matrix of all anchor points; The product of the distance and the normalized eigenvalue is calculated, and the result of subtracting the product from the distance is taken as the distance between the eigenvector and the center vector of the cluster.

[0035] Optionally, returning the archive in the cluster with the smallest distance to the user is specifically: The distance between the feature vector of the archive in the cluster with the smallest distance and the center vector of the cluster is calculated, the archive in the cluster with the smallest distance is sorted in ascending order of the distance, and the archive is returned to the user according to the sorting result.

[0036] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Figure 5 A schematic diagram of a computing device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. .

Claims

1. A file retrieval method, characterized in that: The method comprises: Extract features from the archives, select multiple subsets from the archives, construct a similarity matrix of the subsets using the extracted features, and calculate the Laplace matrix of the subsets based on the similarity matrix of the subsets to obtain the eigenvalues ​​of the Laplace matrix; Obtaining the number of clusters based on the feature values ​​of all subsets, obtaining anchor points according to the number of clusters and the feature values ​​of the subsets, and clustering the archives using the number of clusters and the anchor points; The search keywords input by the user are converted into feature vectors, the distance between the feature vectors and the center vector of each cluster is calculated, and the archives in the cluster with the smallest distance are returned to the user.

2. The method according to claim 1, characterized in that The number of clusters obtained based on the feature values ​​of all subsets is specifically: Select the subset with the minimum number of eigenvalues ​​among all subsets, and for the remaining subsets, retain the eigenvalues ​​with the minimum number of eigenvalues ​​mentioned above for each subset; For each subset, the difference between adjacent eigenvalues ​​is calculated to obtain a difference sequence, and the average of the difference sequences of all subsets is calculated to obtain an average sequence; The position of the maximum value in the average value sequence is found from the average value sequence, and the sequence number of the position is used as the number of clusters.

3. The method according to claim 1, characterized in that The anchor point is obtained according to the number of clusters and the feature value of the subset, specifically: Calculate the sum of the eigenvalues ​​of the number of clusters in front of each subset, and calculate the union of multiple subsets with the smallest sum of eigenvalues; The difference between the union of multiple subsets with the smallest sum of eigenvalues ​​and the intersection of the remaining subsets is taken as the anchor point set, and each element in the anchor point set is an anchor point.

4. The method according to claim 1, characterized in that The distance between the feature vector and the center vector of each cluster is calculated as follows: Calculate the distance from the eigenvector to the center vector of each cluster and obtain the eigenvalues ​​of the Laplacian matrix of all anchor points; The product of the distance and the normalized eigenvalue is calculated, and the result of subtracting the product from the distance is taken as the distance between the eigenvector and the center vector of the cluster.

5. The method according to claim 1, characterized in that The file in the cluster with the smallest distance is returned to the user, specifically: The distance between the feature vector of the archive in the cluster with the smallest distance and the center vector of the cluster is calculated, the archive in the cluster with the smallest distance is sorted in ascending order of the distance, and the archive is returned to the user according to the sorting result.

6. A file retrieval system, characterized in that: The system comprises: The feature extraction module is used to extract features from the archives, select multiple subsets from the archives, construct a similarity matrix of the subsets using the extracted features, and calculate the Laplace matrix of the subsets based on the similarity matrix of the subsets to obtain the eigenvalues ​​of the Laplace matrix; A clustering module, used to obtain the number of clusters based on the feature values ​​of all subsets, obtain anchor points according to the number of clusters and the feature values ​​of the subsets, and cluster the archives using the number of clusters and the anchor points; The search result return module is used to convert the search keywords input by the user into feature vectors, calculate the distance between the feature vectors and the center vector of each cluster, and return the archives in the cluster with the smallest distance to the user.

7. The system according to claim 6, characterized in that The number of clusters obtained based on the feature values ​​of all subsets is specifically: Select the subset with the minimum number of eigenvalues ​​among all subsets, and for the remaining subsets, retain the eigenvalues ​​with the minimum number of eigenvalues ​​mentioned above for each subset; For each subset, the difference between adjacent eigenvalues ​​is calculated to obtain a difference sequence, and the average of the difference sequences of all subsets is calculated to obtain an average sequence; The position of the maximum value in the average value sequence is found from the average value sequence, and the sequence number of the position is used as the number of clusters.

8. The system according to claim 6, characterized in that The anchor point is obtained according to the number of clusters and the feature value of the subset, specifically: Calculate the sum of the eigenvalues ​​of the number of clusters in front of each subset, and calculate the union of multiple subsets with the smallest sum of eigenvalues; The difference between the union of multiple subsets with the smallest sum of eigenvalues ​​and the intersection of the remaining subsets is taken as the anchor point set, and each element in the anchor point set is an anchor point.

9. The system according to claim 6, characterized in that The distance between the feature vector and the center vector of each cluster is calculated as follows: Calculate the distance from the eigenvector to the center vector of each cluster and obtain the eigenvalues ​​of the Laplacian matrix of all anchor points; The product of the distance and the normalized eigenvalue is calculated, and the result of subtracting the product from the distance is taken as the distance between the eigenvector and the center vector of the cluster.

10. The system according to claim 6, characterized in that The file in the cluster with the smallest distance is returned to the user, specifically: The distance between the feature vector of the archive in the cluster with the smallest distance and the center vector of the cluster is calculated, the archive in the cluster with the smallest distance is sorted in ascending order of the distance, and the archive is returned to the user according to the sorting result.

Images