Text data distributed query method and system based on content similarity

Abstract

The invention discloses a text data distributed query method and system based on content similarity. The method comprises four steps of text feature extraction, data index construction, query processing and subsequent processing. The method comprises the following steps: firstly, extracting features in a text data sliding window through text feature extraction, and converting the features into a feature vector sequence; then, through data index construction, extracting prefix features of a text sub-feature sequence, and respectively constructing spatial indexes on each sub-feature dimension; finally, through query processing, candidate sets are queried and filtered on all the sub-indexes based on the lower bound distance, and through follow-up processing, intersection of the candidate sets is taken to complete large-scale accurate query of the text data. The text data distributed index structure provided by the invention has relatively high spatial ductility, and can play an important role in the fields of big data, data mining, databases and the like.

Description

Technical Field

[0001] The present invention belongs to the fields of big data, database, data mining, etc., and specifically relates to a text data distributed query method and system based on content similarity. Background Art

[0002] Text data similarity query methods mainly include methods based on cosine distance, Jaccard similarity, edit distance, and bag-of-words. Cosine similarity is a commonly used text similarity measurement method. It measures the similarity between texts by calculating the cosine value of the angle between two text vectors. Jaccard similarity determines similarity by calculating the number of elements in the intersection of two sets divided by the number of elements in the union. This method regards text data as a vocabulary set. Although the calculation is simple, the accuracy is insufficient. Edit distance measures the similarity between two strings by calculating the minimum number of editing operations (insertion, deletion, replacement, etc.) required to convert one string into another. This method is mainly suitable for text exact matching and difference analysis, and it is difficult to support index construction and efficient similarity query. Bag-of-words-based methods, such as TF-IDF, also regard text as a vocabulary set and compare text similarity by counting word frequencies or using methods such as TF-IDF. However, this method ignores the order of words and only considers the frequency of word occurrence, so the accuracy of semantic similarity measurement is poor. These methods have their own characteristics and are suitable for different application scenarios. For example, cosine similarity and bag-of-words-based methods are more suitable for similarity queries on large-scale text data, while Jaccard similarity and edit distance are more suitable for exact matching and difference analysis. However, it is difficult for the above methods to take into account both large-scale and accurate queries. Summary of the invention

[0003] In order to solve the shortcomings of the prior art and achieve the purpose of efficient query of large-scale text databases, the present invention adopts the following technical solutions:

[0004] The text data distributed query method based on content similarity includes the following steps:

[0005] (1) Text feature extraction, converting text data into a set of multi-dimensional word embedding vector sequences;

[0006] (2) Data index construction: Separate the group of multi-dimensional word embedding vector sequences based on each dimension to obtain the sub-feature sequence of each dimension to construct the index;

[0007] (3) Query processing: converting the query text data into a set of multi-dimensional query text word embedding vector sequences, separating the set of multi-dimensional query text word embedding vector sequences by each dimension, obtaining the query text sub-feature sequence of each dimension, calculating the similarity based on the query text sub-feature sequence and the text data sub-feature sequence on the index, and taking the text objects contained in the index nodes with high similarity as the intermediate candidate set;

[0008] (4) Subsequent processing: taking the intersection of the intermediate candidate sets queried on the sub-features of each dimension, and performing a similarity comparison between the original text data corresponding to the final candidate set and the query text data to obtain the query result.

[0009] Furthermore, in the step (2), the prefix sequence features of the sub-feature sequences are extracted through the sub-feature sequences of each dimension to construct an index; in the step (3), the prefix sequence features of the query text sub-feature sequences are extracted through the query text sub-feature sequences of each dimension, and the Euclidean distance between the prefix sequence features of the query text data and the prefix sequence features corresponding to each index node is calculated in turn on the index; the Euclidean distance based on the prefix feature is the lower bound distance of the multivariate dynamic time warping distance, which can ensure the completeness of the query result, that is, the result of the query based on the multivariate dynamic time warping distance on the original data must be included in the intermediate candidate set based on the prefix feature Euclidean distance query.

[0010] Furthermore, in step (2), the starting value, final value, maximum value, and minimum value of the equal-length prefix sequence of each sub-feature sequence are extracted as features; in step (3), the starting value, final value, maximum value, and minimum value of the equal-length prefix sequence of each query text sub-feature sequence are extracted as features.

[0011] Furthermore, in step (2), a metric index or a spatial index is constructed on the prefix sequence feature set of each sub-feature sequence in the database of this article.

[0012] Furthermore, in step (3), on the index, the Euclidean distance of the prefix sequence feature of each sub-feature sequence of the query text data to the prefix sequence feature of the text data corresponding to the index node is calculated in turn, the nodes and their subtrees with distance values ​​greater than the query threshold are pruned, and the text objects contained in all leaf nodes with distance values ​​less than or equal to the query threshold are returned as the intermediate candidate set.

[0013] Furthermore, in the step (1), the text data is segmented according to a dictionary and stop words are removed, and then the text data is converted.

[0014] Furthermore, in the step (1), the sliding window length is set, and the word2vec algorithm is used to calculate the word embedding vector for the text fragment in each step window, and the text data is converted into a word embedding vector sequence; in the step (3), the query text data is segmented using sliding windows of equal length, and the word2vec algorithm is used to calculate the word embedding vector of each step window, and the query text data is converted into a word embedding vector sequence.

[0015] Furthermore, in step (4), since the final candidate set obtained is an address value, it is necessary to read the original text data corresponding to the final candidate set in sequence through disk I / O, calculate the multivariate dynamic time warping distance between the word embedding vector sequence of each original text data and the word embedding vector sequence of the query text, and take the text objects that are less than or equal to the query threshold as the query results.

[0016] Furthermore, in step (4), the calculation of the multivariate dynamic time warping distance includes the following sub-steps:

[0017] (4.1) Construct a point-to-point distance matrix of two multivariate time series. Each cell value in the matrix is ​​the Euclidean distance of the sampling vectors at the corresponding position of the two series.

[0018] (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized, and the cumulative sum of the path is used as the final distance measurement value.

[0019] A text data distributed query system based on content similarity is applied to a distributed database. The system includes a text feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for the text feature extraction, the data index construction, the query processing and the subsequent processing. The sub-feature sequences of each dimension are deployed in the distributed database.

[0020] The advantages and beneficial effects of the present invention are:

[0021] 1. The present invention applies the dynamic time warping distance to similarity query of text data, designs a compact lower bound distance for it, can effectively support complete query based on the dynamic time warping distance, and has a high query accuracy;

[0022] 2. The distributed index structure adopted by the present invention can support the distributed storage of large-scale text data. Compared with the traditional single index structure that only supports centralized data storage, the present invention has higher spatial scalability;

[0023] 3. The distributed index structure proposed in the present invention has mutually independent sub-feature indexes and has high scalability for feature extraction scale, thus significantly improving query accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0026] like Figure 1 As shown in FIG. 1 , a distributed query method for text data based on content similarity is provided. Given a corpus containing 1 million patent texts, an efficient content similarity query is performed on it. Specifically, the following steps are included:

[0027] (1) Text feature extraction: Segment the text data according to the dictionary and remove stop words; set the sliding window length to 100, use the word2vec algorithm to calculate the 32-dimensional word embedding vector for the text fragment in each window, and convert the patent text data into a word embedding vector sequence; the word embedding vector sequence of a single patent text is as follows:

[0028]

[0029] Where a represents the feature vector of the patent text, the superscript represents the dimension of the text segment, and the subscript n represents the maximum number of steps that the sliding window slides on the patent text, that is, the number of text segments.

[0030] (2) Data index construction: Separate the sub-feature sequences of the word embedding vector sequence and separate the word embedding vector sequence into 32 sub-feature sequences, as follows:

[0031]

[0032] The starting value, final value, maximum value, and minimum value of the prefix sequence of each sub-feature sequence of equal length (and length 100) are extracted as features, as follows:

[0033]

[0034] Among them, the subscript max indicates that the current feature a is the maximum value of the prefix sequence, the subscript min indicates that the current feature a is the minimum value of the prefix sequence, and l indicates that the current feature a is the final value of the prefix sequence;

[0035] A total of 32 sub-feature sequence prefix features corresponding to 1 million patent texts are obtained. On the prefix feature sets of each sub-feature sequence in the patent text database, metric indexes or spatial indexes are constructed for the prefix feature sets of the 32 sub-feature sequences respectively. In this example, M-tree index is used.

[0036] (3) Query processing: Use sliding windows of equal length to segment the query text, use the word2vec algorithm to calculate the word embedding vector of each window, and convert the query patent text into a 32-dimensional word embedding vector sequence; extract the four-dimensional prefix features of each of the 32 sub-feature sequences of the query patent text, as follows:

[0037]

[0038] Wherein, b represents the feature vector of the query patent text, the superscript represents the dimension of the query patent text segment, the subscript m represents the maximum number of steps that the sliding window slides in the query patent text, that is, the number of text segments, the subscript max represents the maximum value of the current feature b as the prefix sequence, the subscript min represents the minimum value of the current feature b as the prefix sequence, and l represents the final value of the current feature b as the prefix sequence;

[0039] In the 32 sub-feature indexes of the database, the Euclidean distance from the query feature to each index node is calculated on each sub-feature sequence index in turn, and the nodes and their subtrees with distance values ​​greater than the query threshold are pruned, and the patent text objects contained in all leaf nodes with distance values ​​less than or equal to the query threshold are returned as the intermediate candidate set;

[0040] The Euclidean distance based on prefix features is the lower bound distance of the multivariate dynamic time warping distance, which can ensure the completeness of the query results, that is, the results of the query based on the multivariate dynamic time warping distance on the original data must be included in the intermediate candidate set of the Euclidean distance query based on the prefix feature.

[0041] (4) Subsequent processing: Take the intersection of the 32 intermediate candidate sets queried on each sub-feature dimension to obtain the final candidate set, read the original patent text data of the final candidate objects in turn through disk I / O, calculate the multivariate dynamic time warping distance between the 32-dimensional word embedding vector sequence of each patent text and the 32-dimensional word embedding vector sequence of the query patent text, and return the patent text objects that are less than or equal to the query threshold as the result. The calculation of the multivariate dynamic time warping distance specifically includes the following sub-steps:

[0042] (4.1) Construct a point-to-point distance matrix of two multivariate time series, where each cell value is the Euclidean distance between the sampling vectors at the corresponding positions of the two series;

[0043] (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized, and the cumulative sum of the path is used as the final distance measurement value.

[0044] A distributed query system for text data based on content similarity is applied to distributed databases. The system includes a text feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for text feature extraction, data index construction, query processing and subsequent processing. The sub-feature sequences of each dimension are deployed in the distributed database.

[0045] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed query method for text data based on content similarity, characterized in that The steps include: (1) Text feature extraction, converting text data into a set of multi-dimensional word embedding vector sequences; (2) Data index construction: Separate the group of multi-dimensional word embedding vector sequences based on each dimension to obtain the sub-feature sequence of each dimension to construct the index; (3) Query processing: converting the query text data into a set of multi-dimensional query text word embedding vector sequences, separating the set of multi-dimensional query text word embedding vector sequences by each dimension, obtaining the query text sub-feature sequence of each dimension, calculating the similarity based on the query text sub-feature sequence and the text data sub-feature sequence on the index, and taking the text objects contained in the index nodes with high similarity as the intermediate candidate set; (4) Subsequent processing: taking the intersection of the intermediate candidate sets queried on the sub-features of each dimension, and performing a similarity comparison between the original text data corresponding to the final candidate set and the query text data to obtain the query result.

2. The text data distributed query method based on content similarity according to claim 1 is characterized in that: In the step (2), the prefix sequence features of the sub-feature sequences are extracted through the sub-feature sequences of each dimension to construct an index; in the step (3), the prefix sequence features of the query text sub-feature sequences are extracted through the query text sub-feature sequences of each dimension, and the Euclidean distance between the prefix sequence features of the query text data and the prefix sequence features corresponding to each index node is calculated in turn on the index to determine the similarity.

3. The text data distributed query method based on content similarity according to claim 2 is characterized in that: In the step (2), the starting value, final value, maximum value, and minimum value of the prefix sequence of equal length of each sub-feature sequence are extracted as features; in the step (3), the starting value, final value, maximum value, and minimum value of the prefix sequence of equal length of each query text sub-feature sequence are extracted as features.

4. The text data distributed query method based on content similarity according to claim 2 is characterized in that: In the step (2), a metric index or a spatial index is constructed on the prefix sequence feature set of each sub-feature sequence in the database of this article.

5. The text data distributed query method based on content similarity according to claim 2 is characterized in that: In the step (3), on the index, the prefix sequence feature of each sub-feature sequence of the query text data and the Euclidean distance of the prefix sequence feature of the text data corresponding to the index node are calculated in turn, the nodes and their subtrees with distance values ​​greater than the query threshold are pruned, and the text objects contained in all leaf nodes with distance values ​​less than or equal to the query threshold are returned as the intermediate candidate set.

6. The text data distributed query method based on content similarity according to claim 1 is characterized in that: In the step (1), the text data is segmented according to a dictionary and stop words are removed, and then the text data is converted.

7. The text data distributed query method based on content similarity according to claim 1 is characterized in that: In the step (1), the sliding window length is set, the word embedding vector is calculated for the text fragment in each step window using the word2vec algorithm, and the text data is converted into a word embedding vector sequence; in the step (3), the query text data is segmented using sliding windows of equal length, the word embedding vector of each step window is calculated using the word2vec algorithm, and the query text data is converted into a word embedding vector sequence.

8. The text data distributed query method based on content similarity according to claim 1 is characterized in that: In the step (4), the original text data corresponding to the final candidate set is read in sequence through disk I / O, the multivariate dynamic time warping distance between the word embedding vector sequence of each original text data and the word embedding vector sequence of the query text is calculated, and the text objects that are less than or equal to the query threshold are taken as query results.

9. The text data distributed query method based on content similarity according to claim 8 is characterized in that: In the step (4), the calculation of the multivariate dynamic time warping distance includes the following sub-steps: (4.1) Construct a point-to-point distance matrix of two multivariate time series. Each cell value in the matrix is ​​the Euclidean distance between the sampling vectors at the corresponding position of the two series. (4.2) Use the dynamic programming algorithm to find the optimal curved path on the point-to-point distance matrix so that the cumulative sum of the element values ​​on the path is minimized, and the cumulative sum of the path is used as the final distance measurement value.

10. A distributed query system of the text data distributed query method based on content similarity according to claim 1, applied to a distributed database, characterized in that: The system includes a text feature extraction module, a data index construction module, a query processing module and a subsequent processing module, which are respectively used for the text feature extraction, the data index construction, the query processing and the subsequent processing. The sub-feature sequences of each dimension are deployed in the distributed database.

Images