Method and system for quickly retrieving similar texts

Abstract

The invention discloses a similar text quick retrieval method and system, and belongs to the technical field of natural language processing and information retrieval, and the similar text quick retrieval method comprises the following specific steps: step 1, vectorizing semantics, and storing the vectorized semantics into a database; using a pre-training language model to respectively encode the query text and the documents in the text library into low-dimensional semantic vectors with fixed dimensions; step 2, hierarchical index retrieval: N candidate documents closest to the query vector are retrieved through a first-level index structure to form a first candidate set; according to the method, millisecond coarse screening is realized through parallel design of the HNSW graph index and the semantic cluster inverted index, the semantic cluster inverted index reduces a search space to a local optimal domain, global precision loss is avoided, a newly added text only needs one-time cluster center distance calculation, and index updating complexity is greatly reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing and information retrieval, and particularly relates to a similar text fast retrieval method and system. BACKGROUND

[0002] The similar text fast retrieval system is an intelligent information retrieval tool based on text content similarity analysis, and the core goal is to quickly locate similar texts to a target text (such as a sentence, a paragraph, or a document) in semantics, structure, or content from a large-scale text library, and return the results in an efficient and accurate manner. It is different from the traditional keyword-based exact match retrieval system, and pays more attention to capturing the deep association between texts.

[0003] However, the traditional keyword matching cannot capture semantic similarity, resulting in the "lexical gap" problem. The retrieval based on word embedding does not consider the sentence-level semantics, and the retrieval speed decreases linearly with the increase of data volume. The semantic similarity calculation based on BERT has high accuracy, but the time complexity of direct full comparison is O(n 2 ), which is difficult to respond in real time. The existing approximate nearest neighbor search such as HNSW and FAISS mainly optimizes the vector distance calculation, and does not combine the text characteristics to optimize the index update and query logic. SUMMARY

[0004] The technical problem to be solved by the application is to overcome the shortcomings of the prior art and provide a similar text fast retrieval method and system.

[0005] The technical solution adopted to solve the above technical problem is as follows: a similar text fast retrieval method, comprising the following specific steps:

[0006] Step 1: semantic vectorization, using a pre-trained language model to encode the query text and the documents in the text library into fixed-dimension low-dimensional semantic vectors respectively;

[0007] Step 2: hierarchical index retrieval, retrieving the N closest candidate documents to the query vector through a one-level index structure to form a first candidate set;

[0008] Step 3: locating the nearest semantic cluster to which the query vector belongs through a two-level index structure, and extracting the M documents with the smallest distance to the cluster center from the inverted list of the semantic cluster to form a second candidate set;

[0009] Step 4: hybrid reordering, for each document in the second candidate set, calculating the hybrid similarity score between the document and the query text; the score is the weighted sum of the semantic similarity and the keyword similarity, wherein the semantic similarity is calculated by the cosine value of the vector angle, and the keyword similarity is calculated by the Jaccard similarity coefficient of the document keyword set;

[0010] Step five: output the top K documents in descending order of scores.

[0011] By the above technical solution, the synonym and context association are captured, the "lexical gap" problem of traditional keyword retrieval is solved, the parallel design of HNSW graph index and semantic cluster inverted index is implemented to realize millisecond-level coarse screening, the search space is reduced to a local optimal domain by the semantic cluster inverted index to avoid global precision loss, and the index update complexity is greatly reduced by only one cluster center distance calculation for new text.

[0012] Further, the primary index structure is a vector index based on a navigable small-world graph, and dynamic updating of the primary index structure includes calculating a dynamic heat weight according to a historical access frequency and a last access time of a document, and when the heat weight is higher than a set threshold, a connection edge with a high-weight node is preferentially established in the index graph.

[0013] By the above technical solution, the graph structure is dynamically optimized according to the access mode to avoid the performance being dragged down by cold data.

[0014] Further, the incremental updating of the secondary index structure includes, when a new document is added, calculating the Euclidean distance between the semantic vector of the new document and all cluster centers, assigning the new document to the semantic cluster closest to the new document, and only updating the inverted list of the cluster.

[0015] By the above technical solution, the inverted list is stored in a distributed manner, and the candidate set is extracted in parallel to improve the data throughput.

[0016] Further, the pre-trained language model is compressed by a knowledge distillation technology, and specifically satisfies: the teacher model is a complete BERT architecture, and the student model is a distillation architecture with half the number of layers; the training loss function is composed of the weighted sum of the mean square error of the output vectors of the teacher-student model and the cross entropy of the prediction results of the student model, and the weight coefficient of the mean square error term is between 0.5 and 0.8.

[0017] By the above technical solution, the model size is greatly reduced, so that the compressed model can run on a mobile device and be extended to an edge computing scenario.

[0018] A system of a similar text fast retrieval method includes a semantic encoding module, a dynamic index engine, an incremental updating service, and a reordering module; the semantic encoding module is responsible for performing a semantic vectorization step; the dynamic index engine includes a primary graph index unit and a secondary semantic cluster inverted index unit in parallel; the incremental updating service is responsible for performing cluster assignment and index updating; and the reordering module is responsible for calculating a hybrid similarity score and sorting.

[0019] By the above technical solution, the calculation accuracy of semantic similarity can be improved, the time complexity of full-amount comparison can be reduced, and real-time response can be realized.

[0020] Further, the dynamic index engine comprises a hotness weight calculation unit configured to update the node weight in real time according to the document access frequency and a time decay factor, and when the node weight rises, a connection edge between the node and a high-weight neighbor node in the navigable small-world graph is preferentially established, wherein the neighbor node needs to satisfy that the Euclidean distance with the current node is less than a set threshold.

[0021] Through the above technical solution, only hot data retains high connectivity, greatly reduces memory occupation, supports real-time data storage and retrieval of streaming data, and avoids downtime maintenance of traditional index global reconstruction.

[0022] The beneficial effects of the present application are as follows: the present application realizes millisecond-level coarse screening through the parallel design of HNSW graph index and semantic cluster inverted index, the semantic cluster inverted index reduces the search space to a local optimal domain, avoids global precision loss, and only needs to calculate the cluster center distance once for new text, greatly reducing the index update complexity. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is the system architecture diagram of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0025] As shown in Figure 1 , the similar text fast retrieval method and system of the present embodiment comprises the following specific steps:

[0026] Step one: semantic vectorization, using a pre-trained language model to encode the query text and the documents in the text library into fixed-dimension low-dimensional semantic vectors respectively;

[0027] Step two: hierarchical index retrieval, retrieving the N candidate documents closest to the query vector through a one-level index structure to form a first candidate set;

[0028] Step three: locating the nearest semantic cluster to which the query vector belongs through a two-level index structure, and extracting the M documents with the smallest cluster center distance from the inverted list of the semantic cluster to form a second candidate set;

[0029] Step four: hybrid reordering, calculating the hybrid similarity score of each document in the second candidate set with the query text; the score is the weighted sum of semantic similarity and keyword similarity, wherein the semantic similarity is calculated by the vector angle cosine value, and the keyword similarity is calculated by the Jaccard similarity coefficient of the document keyword set;

[0030] Step five: finally output the top K documents in descending order of score, capture synonyms, context association, solve the "lexical gap" problem of traditional keyword search, achieve millisecond-level coarse screening through the parallel design of HNSW graph index and semantic cluster inverted index, reduce the search space to the local optimal domain through semantic cluster inverted index, avoid global precision loss, and only need to calculate the cluster center distance once for new text, greatly reducing the index update complexity.

[0031] The primary index structure is a vector index based on a navigable small-world graph, and dynamic updating of the primary index structure includes calculating a dynamic heat weight according to a historical access frequency and a last access time of a document, when the heat weight is higher than a set threshold, a connection edge between the node and a high-weight node is preferentially established in the index graph, and the graph structure is dynamically optimized according to an access mode, so that cold data does not drag the overall performance.

[0032] The incremental updating of the secondary index structure includes, when a new document is added, calculating the Euclidean distance between the semantic vector of the new document and all cluster centers, assigning the new document to the semantic cluster closest to the new document, and only updating the inverted list of the cluster, wherein the inverted list is stored in a distributed manner, candidate sets are extracted in parallel, and data throughput is improved.

[0033] The pre-trained language model is compressed through a knowledge distillation technique, and specifically meets the following conditions: the teacher model is a complete BERT architecture, and the student model is a distillation architecture with half the number of layers; a training loss function is composed of a weighted sum of a mean square error of output vectors of the teacher-student model and a cross-entropy of a prediction result of the student model, wherein the weight coefficient of the mean square error term is between 0.5 and 0.8, which greatly reduces the model size, so that the compressed model can run on a mobile device and be extended to an edge computing scenario.

[0034] A system for a similar text fast retrieval method, comprising a semantic encoding module, a dynamic index engine, an incremental updating service and a reordering module; the semantic encoding module is responsible for performing a semantic vectorization step; the dynamic index engine includes a primary graph index unit and a secondary semantic cluster inverted index unit in parallel; the incremental updating service is responsible for performing cluster assignment and index updating; the reordering module is responsible for calculating a hybrid similarity score and sorting, which can improve the calculation accuracy of semantic similarity, reduce the time complexity of full-amount comparison, and can respond in real time.

[0035] The dynamic index engine includes a heat weight calculation unit configured to update the node weight in real time according to the document access frequency and the time decay factor, and when the node weight rises, a connection edge between the node and a high-weight neighbor node is preferentially established in the navigable small-world graph, wherein the neighbor node needs to satisfy that the Euclidean distance from the current node is less than a set threshold, only hot data retains high connectivity, greatly reduces the memory occupation, supports real-time data storage and retrieval, and avoids the downtime maintenance of traditional index global reconstruction.

[0036] The above merely describes the preferred embodiments of the present application, but is not intended to limit the protection scope of the present application.

Claims

1. A method for fast retrieval of similar texts, characterized in that, The specific steps include the following: Step 1: Vectorize semantics by using a pre-trained language model to encode the query text and documents in the text library into fixed-dimensional low-dimensional semantic vectors. Step 2: Hierarchical index retrieval. The first-level index structure is used to retrieve the N candidate documents that are closest to the query vector, forming the first candidate set. Step 3: Locate the nearest semantic cluster to which the query vector belongs through the secondary index structure, and extract the M documents with the smallest distance from the cluster center from the inverted list of the semantic cluster to form the second candidate set; Step 4: Hybrid Re-ranking. For each document in the second candidate set, calculate its hybrid similarity score with the query text. This score is a weighted sum of semantic similarity and keyword similarity, where semantic similarity is calculated using the cosine of the angle between the vectors, and keyword similarity is calculated using the Jaccard similarity coefficient of the document's keyword set. Step 5: Finally, output the top K documents in descending order of score.

2. The method for fast retrieval of similar texts according to claim 1, characterized in that, The first-level index structure is a vector index based on a navigable small-world graph. The dynamic update of the first-level index structure includes calculating dynamic popularity weights based on the document's historical access frequency and last access time. When the popularity weight is higher than a set threshold, connection edges with high-weight nodes are preferentially established in the index graph.

3. The method for fast retrieval of similar texts according to claim 2, characterized in that, The incremental update of the secondary index structure includes, when a new document is added, calculating the Euclidean distance between its semantic vector and all cluster centers, assigning it to the nearest semantic cluster, and updating only the inverted list of that cluster.

4. The method for fast retrieval of similar texts according to claim 3, characterized in that, The pre-trained language model is compressed using knowledge distillation technology, specifically satisfying the following: the teacher model is a complete BERT architecture, and the student model is a distillation architecture with half the number of layers; the training loss function is composed of the weighted average squared error of the output vectors of the teacher and student models and the cross-entropy of the prediction results of the student model, wherein the weight coefficient of the average squared error term is between 0.5 and 0.

8.

5. The system for a fast similar text retrieval method according to claim 4, characterized in that, It includes a semantic encoding module, a dynamic indexing engine, an incremental update service, and a reordering module; the semantic encoding module is responsible for performing the semantic vectorization step; the dynamic indexing engine includes parallel first-level graph index units and second-level semantic cluster inverted index units; the incremental update service is responsible for performing cluster allocation and index updates; and the reordering module is responsible for calculating and sorting the hybrid similarity scores.

6. The system for a fast similar text retrieval method according to claim 5, characterized in that, The dynamic indexing engine includes a popularity weight calculation unit, which is configured to update node weights in real time based on document access frequency and time decay factor. When a node weight increases, it prioritizes establishing connection edges between the node and high-weight neighboring nodes in the navigable small world graph, where the neighboring nodes must satisfy the condition that the Euclidean distance between them and the current node is less than a set threshold.

Images