A method for similar text retrieval based on vector representation

Abstract

The application discloses a similar text retrieval method based on vector representation. The method first processes the text at the article level by using the word embedding + pooling method, obtains the text vector representation through Bert coding, constructs all text vector representations into a ball tree index by using the ball tree method, performs product quantization on all text vector representations and ball tree center points, and builds a PQ index; during retrieval, the query text is also converted into a vector representation, a distance matrix is calculated through product quantization, KNN retrieval is performed based on the distance matrix and the ball tree index structure, and k similar texts are obtained. The similar text retrieval method based on vector representation constructed by the application supports long text retrieval, can accurately and quickly retrieve massive text vector representations, has good effect on complex or multi-semantic text queries, is very friendly and helpful to users who are not familiar with the jargon of a certain field or want to do exploratory search, and finally, the search is more comprehensive and accurate compared with the IndexIVFPQ method in Faiss.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of similar text retrieval, and particularly relates to a similar text retrieval method based on vector representation. BACKGROUND

[0002] Nowadays, people often use search engines such as Baidu, Sogou and the like in daily life, and these search engines are all keyword-based search engines in principle. The keyword-based search engine is exemplified by a simplified Elasticsearch. Elasticsearch uses a tokenizer to split a document into multiple keywords (i.e. meaningful text units), and then the keywords are mapped to a digital sequence and used to build an inverted index. When searching, Elasticsearch also splits the user's query text into multiple keywords, and then finds which documents contain the user's query and scores their relevance, and finally obtains the ranking. Elasticsearch works well in most cases, but the keyword-based search engine usually encounters the following problems: long text retrieval is not supported, such as paper abstracts or paragraph queries in blogs; the query effect is not good for complex or multi-semantic texts; and it is not friendly to users who are not familiar with the jargon of the field or want to do exploratory search.

[0003] Therefore, the application hopes to have a vector representation search engine that focuses on the context semantics of words.

[0004] In the vector-based search engine, Faiss is a relatively mature vector approximate neighbor search library, which provides efficient similarity search and clustering for dense vectors and supports the search of one billion vectors. The core principles of Faiss are two, inverted index IVF and product quantization PQ. These two methods are the main means for Faiss to achieve high speed, low memory and accurate retrieval. However, using Faiss will face the following problems: Faiss itself does not have the function of converting text into vectors; for a large amount of text, the IVF method in the core principles of Faiss requires the user to set the search cluster number by himself, instead of automatically selecting the most suitable search cluster number by the machine, which may not be comprehensive enough for the search, resulting in missing the optimal solution.

[0005] Therefore, the technical problems of not focusing on the context semantics of words in the above keyword-based search engine and the IVF method of Faiss not being comprehensive are urgent to be solved. SUMMARY

[0006] In view of the deficiencies of the prior art, the application provides a similar text retrieval method based on vector representation, and the specific technical solutions are as follows:

[0007] A similar text retrieval method based on vector representation, the method comprises the following steps:

[0008] (1) The text is divided into sentences, and the adjacent sentences are placed in a segment according to the text order by punctuation and splicing, so that the length of each segment is not more than 510 characters, and [CLS] is added in front of each segment; Finally, the whole text is divided into multiple segments to obtain a segment list;

[0009] (2) The segment list obtained by sentence segmentation is encoded based on BERT, and the vector representation of each segment in the segment list is obtained by BERT learning, wherein the representation of the first [CLS] is used as the vector representation of the segment, and finally the segment vector representation list is obtained;

[0010] (3) The Max-Pooling method is used on all [CLS] vectors in the segment vector representation list obtained in step (2), and the maximum value of each latitude is integrated into a vector, and the vector representation after pooling represents the vector representation of the whole chapter text;

[0011] (4) The idea of ball tree is used to divide all chapter text vector representations into a super sphere, which is represented by a node, and the data points point to a node;

[0012] (5) Based on the ball tree obtained in step (4), the product quantization method of the similar vector retrieval library Faiss is used to compress and encode all chapter text vector representations and all center points of the ball tree, that is, to cluster and quantize the vector representation and construct the PQ index;

[0013] (6) KNN retrieval based on the index obtained in step (5):

[0014] (6.1) The query text is converted into query text vector representation by the method of steps (1)-(3);

[0015] (6.2) The 128-dimensional vector representation is divided into 4 segments of 32-dimensional vector representation by the same method in product quantization, and then the distance between each segment vector and the cluster center in the pre-trained cluster center table is calculated to obtain a 1024*4 distance matrix table, which represents the distance between the query text vector and the vector in the PQ index;

[0016] (6.3) From the ball tree root node, find the nearest leaf node to the query text vector from top to bottom, find the nearest k data points to the query text vector from the leaf node, and store them in a KNN cache queue. Form a query text hypersphere with the query text vector as the center point and the distance from the query text vector to the kth data point in the KNN cache queue as the radius;

[0017] (6.4) Backtracking to check whether the hypersphere corresponding to the sibling node of the current node intersects with the query text hypersphere; the sibling node is another node belonging to the same parent node; according to the triangle theorem, judge whether the two hyperspheres intersect;

[0018] If it is found that the hypersphere corresponding to a sibling node intersects with the query text hypersphere, and the sibling node is a leaf node, calculate whether there is a data point in the node that is closer to the query text vector than the data point in the KNN cache queue. If so, update the KNN cache queue and the radius of the query text hypersphere. Finally, until the root node is backtracked, the retrieval is ended, and the k data points in the KNN cache queue are output as the k nearest neighbor similar texts of the query text.

[0019] Specifically, the method of the ball tree in step (4) is: the ball tree starts with only one root node, and all data points point to it. After each cycle, each node is divided into two child nodes.

[0020] Further, the specific division process of each cycle is:

[0021] (4.1) Calculate the center point of all data points in the node, i.e., the centroid;

[0022] (4.2) Find the data point farthest from the center point as the left child node, and find the data point farthest from the left child node as the right child node;

[0023] (4.3) Let the data points close to the left child node point to the left child node, and otherwise point to the right child node, thereby dividing the new two regions;

[0024] (4.4) Calculate the center point and radius of the two regions to form two hyperspheres; according to the above division process, the cycle is ended when the number of data points in the divided hypersphere is less than the set value, and the ball tree index is finally constructed.

[0025] Further, the clustering and quantization of step (5) are as follows:

[0026] (5.1) Clustering: Assuming that the dimension of each vector representation is 128, each vector is cut into 4 segments, and 4 32-dimensional vector representations are obtained; then, each small vector is clustered, and assuming that the number of clusters is 1024, 1024 cluster centers are obtained, each cluster center is given an ID, and finally, 4 cluster center tables with 1024 cluster centers are obtained;

[0027] (5.2) Quantization: The cluster center table obtained in the first step is used to find the nearest cluster center ID of each 4-segment cut vector representation in the 4 cluster center tables, and the corresponding 4 IDs are assigned to the 4-segment cut vector representation, so that a 128-dimensional vector becomes a vector composed of 4 IDs; finally, the radius of the hyper-sphere of all center nodes of the ball tree is recalculated, and the PQ index is constructed.

[0028] Further, the judgment formula for judging whether two hyper-spheres intersect according to the triangle theorem in step (6.4) is:

[0029] (radius(p) + r) <= |center(p) - q|

[0030] where p is the hyper-sphere corresponding to the brother node of the current node, radius(p) is the radius of the hyper-sphere, center(p) is the center point of the hyper-sphere, q is the query text vector, r is the radius of the query text hyper-sphere, and |center(p) - q| is the distance between the center point p and q;

[0031] If the distance between the center points of two hyper-spheres is less than or equal to the sum of the radii of the two hyper-spheres, it is judged that the two hyper-spheres intersect; if the distance between the center points of two hyper-spheres is greater than the sum of the radii of the two hyper-spheres, it is judged that the two hyper-spheres do not intersect.

[0032] The beneficial effects of the present application are:

[0033] The vector representation-based similar text retrieval method constructed in the present application supports long text retrieval, and compared with the traditional keyword-based retrieval method, the input query content can be more comprehensive; it has good effect on complex or multi-semantic text queries; at the same time, it is very friendly and helpful to users who are not familiar with the jargon of a certain field or want to do exploratory search, especially when the user cannot fully express the professional terms; finally, compared with the IndexIVFPQ method in Faiss, the search is more comprehensive and accurate, and the accuracy is not affected by the artificially specified search cluster number. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 The figure is a step diagram of the vector representation-based similar text retrieval method of the present application;

[0035] Figure 2The specific steps of the search module of the application are shown in the schematic diagram.

[0036] Figure 3 The corresponding ball tree generation diagram is shown. DETAILED DESCRIPTION

[0037] The method of the application will be further described in detail below in combination with the drawings and specific examples, and the purposes and effects of the application will become more apparent. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0038] The application provides a similar text search method based on vector representation, which comprises the following steps:

[0039] (1) The text is divided into sentences, and adjacent sentences are placed in segments in the order of the text by means of punctuation and re-splicing, so that the length of the text in each segment does not exceed 510 characters, and a [CLS] mark is added in front of each segment. Finally, the entire text is divided into multiple segments to obtain a segment list;

[0040] (2) The segment list obtained by dividing the text into sentences is encoded based on BERT, and the vector representation of each segment in the segment list is obtained by using BERT learning. The BERT model uses the representation of the first [CLS] as the vector representation of the segment, and finally obtains a segment vector representation list;

[0041] (3) The Max-Pooling method is used to take the maximum value of each latitude on all [CLS] vectors in the segment vector representation list obtained in step (2), and integrate it into a vector. The vector representation after pooling represents the vector representation of the entire text;

[0042] (4) The idea of ball tree is used to divide all the text vector representations into a super-sphere, which is represented by a node, and the data points point to a node;

[0043] (5) Based on the ball tree obtained in step (4), the product quantization method of the similar vector search library Faiss is used to compress and encode all the text vector representations and all the center points of the ball tree, i.e. to construct a PQ index after clustering and quantizing the vector representation;

[0044] (6) KNN search based on the index obtained in step (5):

[0045] EMBODIMENT

[0046] As a text retrieval method, a large amount of text is needed to build an index, and it is assumed that there is text as shown in Table 1 in a large amount of text, such as the content of serial number a indicating an article about BERT, with a total length of 5000.

[0047] Table 1

[0048]

[0049] As Figure 1 shown, a similar text retrieval method based on vector representation includes the following steps:

[0050] 1. Sentence segmentation for chapter-level text

[0051] For chapter-level text, adjacent sentences are placed in segments in text order by means of punctuation and re-splicing, so that there are enough sentences in each segment but the length does not exceed 510, and a [CLS] identifier is added before each segment. Finally, the entire text is split into multiple segments to obtain a segment list, which is stored in List, as shown in Table 2, for example, the content of serial number a indicates that the first segment of this article about BERT is 503 long, the second segment is 505 long, and the last segment is 410 long.

[0052] Table 2

[0053] Serial number Content Total length a BERT_List [503, 505,..., 410] 5000 b Ball tree_List [510, 508,..., 450] 3000 c Faiss_List [507, 504,..., 310] 10000 d PQ_List [509, 510,..., 430] 4000 e Elasticsearch_List [505, 508,..., 270] 12000 f Lucene_List [506, 507,..., 210] 15000

[0054] 2. BERT encoding

[0055] The segment list obtained after sentence segmentation is encoded based on BERT, and the vector representation of each segment in the segment list is obtained by learning BERT, wherein the BERT model uses the representation output of the first [CLS] as the vector representation of the segment, which can be used for downstream classification tasks and is also helpful for text retrieval tasks. Finally, a segment vector representation list is obtained, where each vector is 768-dimensional, as shown in Table 3, for example, the content of serial number a indicates that each segment in Table 2 serial number a is encoded based on BERT to obtain a 768-dimensional vector, where 0.32 is the first dimension data of the first segment, and 0.88 represents the 758th dimension data of the first segment.

[0056] Table 3

[0057] Serial number Content a BERT_List_BERT [[0.32,..., 0.88], [0.33,..., 0.70],..., [0.40,..., 0.80]] b Ball tree_List_BERT [[0.55,..., 0.71], [0.51,..., 0.75],..., [0.57,..., 0.69]] c Faiss_List_BERT [[0.10,..., 0.11], [0.12,..., 0.20],..., [0.15,..., 0.18]] d PQ_List_BERT [[0.20,..., 0.20], [0.13,..., 0.22],..., [0.16,..., 0.19]] e Elasticsearch_List_BERT [[0.78,..., 0.66], [0.68,..., 0.73],..., [0.90,..., 0.86]] f Lucene_List_BERT [[0.90,..., 0.71], [0.95,..., 0.89],..., [0.75,..., 0.69]]

[0058] 3. Pooling method

[0059] Using the Max-Pooling method, the maximum value of each latitude is taken on all [CLS] vectors in the segment vector representation list and integrated into a vector. The pooled representation can represent the vector representation of the entire text. The formula of the Max-Pooling method is:

[0060] [C0, C1, C2,..., C 767 ] = [Max{c 1i ,c 2i ,c 3i ,...,c ni}], 0≤i≤767

[0061] Where n is the length of the segment vector representation list. The Max-Pooling method is applied to the content of Table 3 to obtain the results shown in Table 4. For example, the content of sequence number a represents the pooled overall text vector representation of the article about BERT, where 0.40 is the maximum value of the first dimension data in the segment vector representation list of sequence number a in Table 3, and 0.88 is the maximum value of the 768th dimension data.

[0062] Table 4

[0063] Serial number Content a BERT_pooling [0.40,..., 0.88] b Ball tree_pooling [0.57,..., 0.75] c Faiss_pooling [0.15,..., 0.20] d PQ_pooling [0.20,..., 0.22] e Elasticsearch_pooling [0.90,..., 0.86] f Lucene_pooling [0.95,..., 0.89]

[0064] As shown in Figure 3 , the idea of ball tree is used to divide all text vector representations into hyper-spheres to form a ball tree, where a hyper-sphere is represented by a node, and the data points point to the nodes.

[0065] The ball tree initially has only one root node, and all data points point to it. In each subsequent loop, each node is divided into two child nodes. The division process in each loop is as follows: first, calculate the center point of all data points in the node, i.e. the centroid; second, find the data point farthest from the center point as the left child node, and find the data point farthest from the left child node as the right child node; third, let the data points close to the left child node point to the left child node, otherwise point to the right child node, and divide the new two regions; fourth, calculate the center point and radius of the two regions to form two hyper-spheres. According to the above division process, the loop is repeated until the number of data points in the divided hyper-spheres is less than the set value, and the ball tree index is finally constructed.

[0066] With Figure 2For example, the center node of the region where data points a, b, c, d, e, f are located is A, and all data points point to A. First, find the data point e farthest from the center point as the left child node C of A. Then, find the data point b farthest from C as the right child node B of A. All data points pointing to A are directed to C if they are close to C, otherwise to B, such as Figure 2 In the middle, a, b point to C, and c, d, e, f point to B. In this way, two new regions are divided, and the center points and radii of the two regions are calculated. Then the above division is repeated for the new regions, and the position of each center node and the radius of the region are recorded, and finally a ball tree and a hyper-sphere are obtained.

[0067] 4. Product Quantizer method

[0068] The Product Quantizer method of the Faiss library of similar vector retrieval is used to compress and encode all chapter text vector representations and all center points of the ball tree, mainly divided into two steps, clustering and quantization.

[0069] First, clustering: assuming that the dimension of each vector representation is 128, each vector is divided into 4 segments, obtaining 4 segments of 32-dimensional vector representations. Then, each small vector is clustered respectively, assuming that the number of clusters is 1024, then 1024 cluster centers are obtained, and each cluster center is assigned an ID, and finally 4 cluster center tables with 1024 cluster centers are obtained.

[0070] Second, quantization: through the cluster center table obtained in the first step, find the nearest cluster center ID of the 4 segment divided vector representation of each vector representation in the 4 cluster center tables, and assign the corresponding 4 IDs to the 4 segment divided vector representation. In this way, a 128-dimensional vector becomes a vector composed of 4 IDs. Finally, the radius of the hyper-sphere where all center nodes of the ball tree are located is recalculated, and the PQ index is constructed. Assuming that the encoding of the text in table 4 is shown in table 5, for example, the content of serial number a represents the content of serial number a after Product Quantizer in table 4, where 40 represents the cluster center ID of the first cluster center table as 40, and the radius 79 represents the new radius of the hyper-sphere where a is located.

[0071] Table 5

[0072] Serial number Content a BERT_PQ [40, 601, 500, 88], radius: 79 b Ball tree_PQ [57, 321, 621, 75], radius: 120 c Faiss_PQ [15, 856, 376, 20], radius: 95 d PQ_PQ [20, 834, 333, 22], radius: 134 e Elasticsearch_PQ [90, 666, 464, 86], radius: 101 f Lucene_PQ [95, 778, 515, 89], radius: 142

[0073] 5. Retrieval query text method

[0074] KNN retrieval: the first step, the query text is converted into a query text vector representation by the method of steps 1), 2), 3); the second step, the 128-dimensional vector representation is divided into four 32-dimensional vector representations in the same way as in the Product Quantizer, and then the distance between each segment vector and the cluster center in the pre-trained cluster center table is calculated to obtain a 1024*4 distance matrix table, and the distance between the query text vector representation and the vector in the index is calculated; the third step, starting from the root node of the ball tree, the nearest leaf node to the query text vector is found from top to bottom, and the k nearest data points to the query text vector are found from the leaf node and stored in a KNN cache queue, and a query text hypersphere is formed with the query text vector as the center point and the distance between the query text vector and the kth data point in the KNN cache queue as the radius; the fourth step, the sibling nodes (another node belonging to the same parent node) of the current node are checked to see if the hyperspheres corresponding to the sibling nodes intersect with the query text hypersphere. The formula for judging whether two hyperspheres intersect is:

[0075] (radius(p)+r)≤|center(p)-q|

[0076] In the formula, p is the hypersphere corresponding to the sibling node of the current node, radius(p) is the radius of the hypersphere, center(p) is the center point of the hypersphere, q is the query text vector, r is the radius of the query text hypersphere, and |center(p)-q| is the distance between the center point p and the query text vector q. The formula applies the triangle theorem.

[0077] If the distance between the center points of two hyperspheres is less than or equal to the sum of the radii of the two hyperspheres, it is determined that the two hyperspheres intersect; if the distance between the center points of two hyperspheres is greater than the sum of the radii of the two hyperspheres, it is determined that the two hyperspheres do not intersect.

[0078] If it is found that the hypersphere corresponding to a sibling node intersects with the query text hypersphere, and the sibling node is a leaf node, it is determined whether there is a data point in the node that is closer to the query text vector than the data points in the KNN cache queue. If so, the KNN cache queue and the radius of the query text hypersphere are updated. Finally, the retrieval is completed when the backtracking reaches the root node, and the k data points in the KNN cache queue are output as the k nearest neighbor similar texts of the query text.

[0079] For those skilled in the art, the technical solutions described in the foregoing examples can be modified, or some technical features can be replaced with equivalents. Any modification, equivalent replacement, etc. within the spirit and principles of the invention shall be included in the protection scope of the invention.

Claims

1. A similar text retrieval method based on vector representation, characterized in that, The method includes the following steps: (1) The text is segmented into sentences. Adjacent sentences are placed into segments in text order by breaking and reassembling the sentences, so that the text length in each segment does not exceed 510 characters. The [CLS] identifier is added before each segment. Finally, the entire text is split into multiple segments to obtain a segment list. (2) The segment list obtained after segmentation is based on BERT encoding. The vector representation of each segment in the segment list is obtained by learning BERT. The BERT model uses the representation output of the first [CLS] as the vector representation of the segment, and finally obtains the segment vector representation list. (3) Using the Max-Pooling method, take the maximum value of each dimension of all [CLS] vectors in the segment vector representation list obtained in step (2) and integrate them into a single vector. The vector representation after pooling represents the vector representation of the entire text. (4) Using the idea of ​​ball trees, all the text vectors of the chapters are divided into hyperspheres as data points to form a ball tree. Each hypersphere is represented by a node, and its data points point to each node. (5) Based on the ball tree obtained in step (4), the product quantization method of the similar vector retrieval library Faiss is used to compress and encode all the center points of the ball tree for all chapter text vector representations, that is, to construct the PQ index after clustering and quantizing the vector representations; (6) Perform KNN retrieval based on the index obtained in step (5): (6.1) The obtained query text is converted into a query text vector representation by the method in steps (1)-(3); (6.2) Using the same method in product quantization, the 128-dimensional vector representation is divided into 4 segments of 32-dimensional vector representation. Then, the distance between each segment of vector and the cluster center in the previously pre-trained cluster center table is calculated to obtain a 1024*4 distance matrix table, which represents the distance between the query text vector and the vector in the PQ index. (6.3) Starting from the root node of the ball tree, find the leaf node closest to the query text vector from top to bottom, find the k data points closest to the query text vector from the leaf nodes, store them in a KNN cache queue, and form a query text hypersphere with the query text vector as the center point and the distance from the query text vector to the kth data point in the KNN cache queue as the radius. (6.4) Backtrack to check if the hypersphere corresponding to the sibling node of the current node intersects with the hypersphere of the query text; the sibling node is another node belonging to the same parent node; determine whether the two hyperspheres intersect according to the triangle theorem; If a sibling node's hypersphere intersects with the query text hypersphere, and that sibling node is a leaf node, calculate whether there are any data points in that node that are closer to the query text vector than the data points in the KNN cache queue. If so, update the radius of the KNN cache queue and the query text hypersphere. Finally, the search ends when the root node is reached, and the k data points in the KNN cache queue are output as the k nearest neighbor similar texts of the query text.

2. The similar text retrieval method based on vector representation according to claim 1, characterized in that, The idea behind the ball tree in step (4) is as follows: initially, the ball tree has only one root node, and all data points point to it. In each subsequent loop, each node is divided into two child nodes.

3. The similar text retrieval method based on vector representation according to claim 2, characterized in that, The specific division process for each cycle is as follows: (4.1) Calculate the centroid of all data points within the node; (4.2) Find the data point farthest from the center point as the left child node, and then find the data point farthest from the left child node as the right child node; (4.3) Make the data points closer to the left child node point to the left child node, otherwise point to the right child node, thus dividing the area into two new regions; (4.4) Calculate the center point and radius of the two regions to form two hyperspheres; repeat the above division process until the number of data points in the divided hyperspheres is less than the set value, and finally construct the ball tree index.

4. The similar text retrieval method based on vector representation according to claim 1, characterized in that, The clustering and quantization in step (5) are specifically as follows: (5.1) Clustering: Assume that each vector has a dimension of 128, and each vector is divided into 4 segments to obtain 4 segments of 32-dimensional vector representation; then cluster each small vector separately, assuming that the number of clusters is 1024, then 1024 cluster centers can be obtained, and each cluster center is assigned an ID, finally obtaining 4 tables with 1024 cluster centers; (5.2) Quantization: Using the cluster center table obtained in the first step, find the nearest cluster center ID corresponding to the 4 segmented vector representation of each vector representation in the 4 cluster center tables, and assign the corresponding 4 IDs to the 4 segmented vector representations. Thus, a 128-dimensional vector becomes a vector composed of 4 IDs. Finally, recalculate the radius of the hypersphere containing all the center nodes of the ball tree, and then construct the PQ index.

5. The similar text retrieval method based on vector representation according to claim 1, characterized in that, The formula for determining whether two hyperspheres intersect based on the trigonometric theorem in step (6.4) is as follows: (radius(p)+r)≤|center(p)-q| Where p is the hypersphere corresponding to the sibling node of the current node, radius(p) is the radius of the hypersphere, center(p) is the center point of the hypersphere, q is the query text vector, r is the radius of the query text hypersphere, and |center(p)-q| is the distance from the center point of p to q. If the distance between the centers of two hyperspheres is less than or equal to the sum of their radii, then the two hyperspheres intersect; if the distance between the centers of two hyperspheres is greater than the sum of their radii, then the two hyperspheres do not intersect.

Images