A method and system for literature semantic search based on elastic search

Abstract

The application discloses a literature semantic search method and system based on elastic search, and relates to data retrieval. A text2vec-base-multilingual model trained based on a CoSENT method is used to perform vectorization processing on a list of noun phrases. According to semantic vectors, approximate nearest neighbor search is performed in a second retrieval module to obtain first candidate data. Query text data is input into the first retrieval module, and keyword matching is performed through a BM25 algorithm to obtain second candidate data. The first candidate data and the second candidate data are fused to obtain third candidate data. A Sequence Matcher algorithm is used to calculate the string similarity between extended words in the third candidate data, a similarity threshold is set based on the length of the longest common subsequence, and deduplication is performed to obtain fourth candidate data. Weight distribution based on position and similarity score is performed on the fourth candidate data, the score interval is expanded to enhance the distinguishability of the extended words, and an extended word recommendation list data is obtained. The application significantly improves the accuracy of literature retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data retrieval, in particular to a literature semantic search method and system based on elastic search. BACKGROUND

[0002] In the field of traditional search engine technology, Apache Lucene, as a representative of open-source full-text retrieval library, realizes efficient text retrieval function through technologies such as inverted index, word segmentation and text analysis, and relevance scoring. Based on Lucene technology, a variety of distributed search engine solutions have emerged. Among them, Solr, as an early distributed search engine, is based on Lucene and supports distributed indexing and querying, and realizes horizontal expansion and high availability through sharding and replication, but its configuration and management are relatively complex, and the real-time performance is not good.

[0003] Elasticsearch (ES), as a new generation of distributed search engine, has made important innovations and improvements in distributed architecture optimization, real-time performance, ease of use and ecological integration on the basis of inheriting Lucene core technology. ES is a distributed, real-time search and analysis engine based on Apache Lucene, mainly used for full-text search, structured data retrieval, log analysis and data visualization scenarios. Its core technologies include inverted index, distributed storage, word segmentation technology and relevance scoring algorithms (such as TF-IDF, BM25) etc. The core features of ES are distributed architecture, real-time performance, efficient full-text search and multi-purpose, which can handle large-scale data and support horizontal expansion. Compared with traditional technology, ES has built-in distributed storage and retrieval functions, supports near-real-time indexing and searching, provides simple and easy-to-use RESTful API, and integrates with Logstash, Kibana and other tools to form a complete log analysis and data visualization solution (ELKStack). These advantages make ES one of the most popular search engines, suitable for large-scale data retrieval and analysis scenarios.

[0004] However, with the rapid development of artificial intelligence and natural language processing technology, the traditional keyword-based search technology gradually exposes its limitations. Modern semantic retrieval technology is based on natural language processing (NLP) technology, such as word vector (Word2Vec, GloVe) and sentence embedding (BERT, Sentence-BERT), which can capture synonyms, near-synonyms and context information, and realize deep semantic understanding.

[0005] Specifically, while Elasticsearch, as a keyword-based search engine, excels in full-text search and distributed processing, its core design inherently limits its semantic retrieval capabilities. Built on Apache Lucene, ES uses inverted indexes and the BM25 (or TF-IDF) algorithm for keyword retrieval. This mechanism enables it to efficiently handle literal matching but fails to understand semantics. For example, ES cannot recognize synonyms or near-synonyms, nor can it capture contextual information, resulting in a query for "AI technology" not matching documents containing "artificial intelligence." Furthermore, ES has limited capabilities in multilingual and complex semantic processing. While its tokenizer and text analyzer support multiple languages, they still rely on rule-based or statistical methods, failing to handle deep semantic information.

[0006] Meanwhile, Elasticsearch's core design does not integrate a semantic model, making it unable to directly support vector-based semantic retrieval. The high computational complexity of semantic retrieval conflicts with Elasticsearch's high-performance, low-latency design goals, resulting in limited performance of Elasticsearch in scenarios requiring deep semantic understanding (such as intelligent question answering and semantic search). This is an inherent limitation of Elasticsearch as a traditional search engine.

[0007] Therefore, how to combine the efficient keyword retrieval capabilities of traditional Elasticsearch with modern semantic retrieval technology to build a hybrid retrieval system that is both high-performance and supports semantic understanding has become an important direction for the development of current search engine technology and a technical problem that urgently needs to be solved. Summary of the Invention

[0008] To address the lack of semantic understanding in Elasticsearch's keyword-based retrieval, this application provides a document semantic search method and system based on Elasticsearch. By constructing a hybrid retrieval architecture that includes a traditional keyword retrieval module and a semantic vector retrieval module, it combines the Apache Lucene BM25 algorithm with the semantic retrieval capabilities of the Milvus vector database. This allows it to maintain the efficient keyword matching performance of traditional search engines while possessing deep semantic understanding and synonym recognition capabilities, significantly improving the accuracy of document retrieval and user experience.

[0009] One aspect of this application provides a document semantic search method based on Elasticsearch, comprising: constructing a hybrid retrieval architecture including a first retrieval module and a second retrieval module; the first retrieval module uses Elasticsearch based on Apache Lucene to process structured text data through an inverted index and the BM25 algorithm; the second retrieval module uses the Milvus vector database to process vector data using a vector index structure; preprocessing the input query text data to generate a list of noun phrases; vectorizing the noun phrase list using a text2vec-base-multilingual model trained based on the CoSENT method to obtain semantic vectors; using the semantic vectors as query vectors, performing an approximate nearest neighbor search in the second retrieval module to obtain extended word data, similarity scores, and retrieval ranking positions as first candidate data; inputting the query text data into the first retrieval module, performing keyword matching retrieval using the BM25 algorithm, calculating statistical scores based on term frequency and inverse document frequency, and retrieval ranking positions as second candidate data; fusing the first and second candidate data, and aggregating the fused candidate data according to logical operators in the query text data to obtain third candidate data; and employing a Sequence... The Matcher algorithm calculates the string similarity between extended words in the third candidate data, sets a similarity threshold based on the length of the longest common subsequence, and performs deduplication to obtain the fourth candidate data. The fourth candidate data is then weighted based on position and similarity scores, and the distinguishability of extended words is enhanced by expanding the score range, resulting in the final extended word recommendation list data.

[0010] Furthermore, a list of noun phrases is generated, including: counting the number of characters N in the query text data; when N is greater than a threshold... When the query text data is corrected, pycorrector, which is based on a statistical language model, is used to perform text correction on the query text data and output the corrected query text data; otherwise, the original query text data is used as the corrected query text data. The value ranges from 3 to 10.

[0011] Determine if the length of the corrected query text data is greater than the threshold. , The value ranges from 2 to 5. If yes, the Spacy en_core_web_trf model based on the Transformer architecture is used to perform lexical and syntactic analysis on the corrected query text data, generating document object data containing part-of-speech tags and grammatical structures; if no, the corrected query text data is used as document object data; based on the document object data, a set of noun phrases is extracted using the doc.noun_chunks method built into the Spacy model; the set data structure is used to remove duplicates from the set of noun phrases to obtain the final list of noun phrases.

[0012] Among them, pycorrector is an automatic Chinese text correction tool based on statistical language models. By analyzing the frequency of word occurrence and contextual relationships in a large-scale corpus, and combining the edit distance algorithm and the n-gram model, it can automatically identify and correct spelling errors, vocabulary errors, and grammatical errors in user input. In this scheme, it is used to preprocess query text with more than the threshold N0 characters to improve the accuracy of input text.

[0013] Spacy en_core_web_trf model: A pre-trained English natural language processing model based on the Transformer architecture, integrating deep learning technologies such as BERT, and possessing powerful lexical analysis, syntactic analysis, named entity recognition, and semantic understanding capabilities. In this scheme, it is used to perform deep language analysis on error-corrected query texts that exceed the threshold C0 in length, generating document object data containing part-of-speech tagging, dependency relations, and grammatical structures.

[0014] Spacy model: An industrial-grade open-source natural language processing framework that provides a complete text processing pipeline, including word segmentation, part-of-speech tagging, named entity recognition, dependency parsing, and other functions. In this solution, it serves as the core language analysis engine, responsible for processing document object data and providing noun phrase extraction functionality.

[0015] The doc.noun_chunks method is a built-in noun phrase extraction method in the Spacy model. Based on dependency parsing and grammatical rules, it can automatically identify and extract noun phrase chunks in text. These noun phrases usually contain core semantic information and are used in this scheme to extract a set of noun phrases that carry the main semantic content from document object data.

[0016] The set data structure is a set data type in Python, implemented based on a hash table. It features element uniqueness and fast lookup, and can perform insertion and lookup operations with an average time complexity of O(1). In this scheme, it is used to automatically deduplicate the extracted set of noun phrases, eliminate duplicate noun phrases, and improve the efficiency of subsequent processing.

[0017] List of noun phrases: The final output obtained after preprocessing, language analysis, extraction and deduplication operations, contains the core semantic elements of the query text. It is a collection of noun phrases organized in list form and serves as the input data for subsequent vectorization processing in this scheme, providing refined semantic units for semantic retrieval.

[0018] In particular, spelling errors in user input can lead to retrieval failures. Furthermore, long text queries make it difficult to extract core semantic information, and repetitive noun phrases can affect retrieval efficiency. Therefore, this application introduces a multi-level preprocessing mechanism that combines pycorrector for automatic error correction and Spacy for deep language analysis.

[0019] Furthermore, semantic vectors are obtained, including: using the CoSENT method, training the text2vec-base-multilingual model with a cosine similarity ranking loss function L; using the trained text2vec-base-multilingual model, semantically encoding the list of noun phrases, and converting each noun phrase into a high-dimensional semantic vector.

[0020] The expression for the sorting loss function L is as follows:

[0021] Where i, j, k, l represent the sample indices in the sample pair. This represents the sentence embedding vector of the corresponding sample. Represents the set of all positive sample pairs. Represents the set of all negative sample pairs; These are boundary parameters used to control the spacing between positive and negative sample pairs; where, .

[0022] Among them, the text2vec-base-multilingual model is a multilingual sentence-level semantic encoding model based on the Transformer architecture. Built upon Transformer architectures such as BERT, it employs a multi-head self-attention mechanism and a feedforward neural network, enabling parallel processing of all word positions in a sequence, capturing long-distance semantic dependencies, and generating dense vector representations of fixed dimensions. Pre-trained on a large-scale multilingual corpus, it supports semantic understanding of multiple languages, including Chinese and English, and possesses cross-lingual semantic alignment capabilities, mapping semantically similar texts from different languages ​​to nearby positions in the vector space.

[0023] In particular, addressing the issue of insufficient sentence-level semantic capture in traditional word vector models, this application employs the CoSENT (Cosine Sentence) contrastive learning method. This method involves supervised learning by constructing a large number of positive and negative sample pairs and using cosine similarity as a semantic similarity metric to train a text2vec-base-multilingual model to learn sentence-level semantic representations. Unlike traditional word-level vector models such as Word2Vec and GloVe, the CoSENT method can holistically understand the semantics of sentences or phrases, capturing complex semantic relationships and contextual dependencies between words. The generated high-dimensional semantic vectors contain complete semantic information of noun phrases, rather than simple word combinations.

[0024] On the other hand, in the loss function L of this application, the cosine similarity of positive sample pairs (sentence pairs with similar semantics) is maximized, while the cosine similarity of negative sample pairs (sentence pairs with dissimilar semantics) is minimized. The distance between positive and negative samples is controlled by the boundary parameter λ. When the similarity of a positive sample pair exceeds the similarity of a negative sample pair plus the boundary λ, the loss is 0; otherwise, gradient optimization is performed. This ranking mechanism forces the model to learn a more accurate semantic boundary, significantly enhancing the discriminative power of positive and negative samples.

[0025] Furthermore, the text2vec-base-multilingual model is trained on a large-scale multilingual corpus, possessing cross-lingual semantic understanding capabilities and able to handle mixed Chinese and English query scenarios. Through the multi-head self-attention mechanism of the Transformer architecture, the model can process all positions in a sequence in parallel, capturing long-distance dependencies, and generating semantic vectors with good semantic continuity and comparability.

[0026] Furthermore, the first candidate data is obtained by inputting the semantic vector as a query vector into the Milvus vector database;

[0027] Perform an approximate nearest neighbor search based on cosine similarity in the Milvus vector database to calculate the similarity score between the query vector and the word vectors stored in the Milvus vector database. Obtain the top 1 candidate extended word data with the highest similarity scores. The candidate extended word data includes the extended word text, similarity score, and ranking position index in the search results. Perform threshold filtering on the top 1 candidate extended word data to determine whether the similarity score of the ranked extended word is lower than a preset threshold. If it is, determine that there is no valid synonym extended word and return empty first candidate data. If not, retain the candidate extended word data that meets the threshold requirement. Combine the filtered candidate extended word data, similarity score, and ranking position index into a triplet as the first candidate data.

[0028] In particular, traditional databases cannot efficiently handle similarity calculations for high-dimensional vectors, resulting in excessively long response times for semantic retrieval of tens of millions of documents. This application addresses this issue by employing the Milvus vector database, utilizing its dedicated vector indexing structures (such as IVF, HNSW, and ANNOY) to replace traditional B+ tree indexes. These vector indexes are optimized for the geometric characteristics of high-dimensional spaces, reducing the linear search complexity from O(n) to O(log n) or even lower. Milvus also employs hardware optimization techniques such as SIMD instruction set acceleration and GPU parallel computing, significantly improving the execution efficiency of vector operations and resolving the performance bottleneck of traditional relational databases when processing high-dimensional vectors.

[0029] On the other hand, this application employs an Approximate Nearest Neighbor (ANN) search strategy instead of an exact brute-force search. The technical principle is to construct a multi-layered graph structure or clustering index to divide the high-dimensional vector space into multiple subspaces. During the search, only a portion of the candidate set needs to be traversed instead of the entire dataset, resulting in a significant performance improvement at the cost of minimal accuracy loss. This is combined with an efficient formula for calculating cosine similarity. Through optimization of vector dot product and modulus pre-computation, millisecond-level similarity calculation response was achieved.

[0030] In addition, this application performs early filtering of search results by setting a similarity threshold. When the similarity score of the first-ranked extended word is lower than the threshold, it is directly determined that there are no valid synonym extended words and an empty result is returned, thus avoiding meaningless calculations and processing in the future.

[0031] Furthermore, the second candidate data is obtained by: inputting the query text data into the Elasticsearch first retrieval module of Apache Lucene; using an inverted index structure to map each word in the query text data to a list of documents containing the corresponding word, establishing a correspondence between words and documents; and calculating the similarity score between the query text data Q and document D based on the BM25 algorithm. ; Obtain the top two matching documents with the highest scores in the BM25 algorithm. The matching document results include document identifier, similarity score, and position index in the index sort; Use the top two matching document results as the second candidate data.

[0032] Among them, Apache Lucene, an open-source full-text search engine library, serves as a core foundational technology in the field of information retrieval, providing complete text indexing and search capabilities. It employs an inverted index data structure to achieve fast text retrieval, supports complex query syntax and Boolean logic combinations, offers multiple text analyzers for word segmentation and text processing, and features efficient disk storage and memory management mechanisms. In this solution, it serves as the underlying engine of the first retrieval module, responsible for handling the indexing of structured text data and keyword matching retrieval.

[0033] Elasticsearch is a distributed, real-time search and analytics engine built on Apache Lucene. It employs a distributed architecture to support horizontal scaling and high availability, provides a RESTful API for easy system integration, supports real-time indexing and near real-time search, possesses powerful aggregation analysis and data visualization capabilities, and has built-in load balancing and fault recovery mechanisms. In this solution, it serves as the concrete implementation of the first retrieval module, encapsulating the complexity of Lucene and providing distributed search capabilities. It is responsible for receiving query text data, performing inverted index operations, running the BM25 algorithm, and returning sorted matching literature results.

[0034] BM25 Algorithm: Best Matching 25 algorithm, a document ranking algorithm based on a probabilistic information retrieval model, calculates the relevance score between a document and a query through a non-linear combination of term frequency (TF) and inverse document frequency (IDF).

[0035] Furthermore, similarity score : Where Q represents the query text data, containing 1 to n query terms. ;D indicates a document; Indicates terms The term frequency (TF) in document D; The word count of document D; This represents the average length of all documents in the document collection; b represents a parameter; The value of is in the range of 1.2 to 2; the value of b is in the range of 0.5 to 0.85. Indicates terms Inverse document frequency (IDF): Where N represents the total number of documents in the document collection; Indicates included terms The number of documents;

[0036] Specifically, on the one hand, regarding the parameters of the traditional BM25 algorithm... The fixed parameter b leads to an inability to adapt to different document set characteristics. This application introduces a parameter range constraint mechanism. By... The parameter is limited to the range of 1.2 to 2, and the b parameter is limited to the range of 0.5 to 0.85, thus realizing the dynamic adjustment capability of the parameters. The parameter controls the word frequency saturation point, and a smaller one... Value (1.2) applies to short document collections; larger ones... The value (2.0) is suitable for long document collections; the b parameter controls the strength of document length normalization, with a smaller b value (0.5) weakening the influence of length and a larger b value (0.85) strengthening length normalization. By dynamically selecting the optimal parameter combination based on statistical characteristics such as the average length and vocabulary distribution of the document collection, the algorithm can adapt to the characteristics of document databases of different domains and sizes.

[0037] On the other hand, regarding traditional inverse document frequency calculation In extreme cases, numerical anomalies may occur. This application employs Laplace smoothing technology to improve the IDF calculation formula to... By performing a smoothing operation by adding 1 to both the numerator and denominator simultaneously, the error when... The division by zero error that occurs at that time prevents when The loss of weights due to an IDF value of 0 ensures that all terms receive a reasonable inverse document frequency weight. This smoothing process is particularly suitable for weight calculation of rare and high-frequency words, improving the numerical stability and robustness of the algorithm.

[0038] Furthermore, the third candidate data is obtained by: acquiring logical operators in the query text data, including the first operator representing "and", the second operator representing "or", and the third operator representing "not"; merging the matching results from the first and second candidate data, extracting the extended words to form a candidate vocabulary set; identifying repeated extended words in the candidate vocabulary set, and aggregating the repeated extended words according to the logical operators: when the logical operator is the first operator "and", the similarity scores of the repeated extended words are accumulated to strengthen the intersection result; when the logical operator is the second operator "or", the similarity scores of the repeated extended words are averaged to balance the union result; when the logical operator is the third operator "not", the corresponding extended words are deleted from the candidate vocabulary set; retaining the original similarity scores of the non-repeating extended words in the first and second candidate data, and merging them with the aggregated repeated extended words; calculating the sorting position index of the merged candidate data, generating a data set containing extended words, similarity scores, and sorting position indices, as the third candidate data.

[0039] In particular, to address the problem that traditional retrieval systems cannot effectively handle complex logical combination queries, this application decomposes complex Boolean queries into executable logical operation sequences by identifying the three logical operators (and, or, not) in the query text.

[0040] To address the issue of a single strategy for aggregating scores of duplicate results, this application designs differentiated aggregation algorithms based on the semantic features of different logical operators. When the logical operator is "AND," an accumulation strategy is adopted. Based on the characteristic that "logical AND" requires multiple conditions to be met simultaneously, the weight of the intersection result is strengthened by accumulating scores, reflecting the confidence of multiple matches. When the logical operator is "OR," an averaging strategy is adopted. Based on the inclusive feature of "logical OR," the average score avoids excessive bias from a single high-scoring result, ensuring the balance of the union result. When the logical operator is "NOT," a deletion strategy is adopted, directly removing exclusion words from the candidate set to achieve precise negative filtering.

[0041] Furthermore, the fourth candidate data is obtained, including: acquiring the extended word text from the third candidate data to form a list of extended words to be processed; and using the Sequence Matcher algorithm to calculate the string similarity ratio between any two words in the extended word list. ,in, This represents the length of the longest common subsequence between strings a and b; and The lengths of strings a and b are represented respectively. The string similarity ratio is checked to see if it is greater than a preset threshold. If it is, deduplication is performed. The expanded words after deduplication and their corresponding similarity scores are combined with the sorting position index to obtain the fourth candidate data.

[0042] Specifically, addressing the issue of numerous highly similar and repetitive extended words in the search results, this application employs the Sequence Matcher algorithm based on the Longest Common Subsequence (LCS) for precise similarity calculation. By using a dynamic programming algorithm to calculate the length len (LCS) of the longest common subsequence between two strings, compared to simple edit distance or character overlap calculations, the LCS algorithm can identify continuous matching segments and sequentially preserved common parts in the strings, more accurately reflecting the structural similarity between words. Furthermore, the LCS algorithm does not require completely continuous character matching, allowing for jumps and intervals, and can identify semantically related word pairs that differ slightly in form, such as "machine learning" and "machine-learning," "AI" and "artificial intelligence," and "deep learning" and "deep neural network learning." By calculating the longest common subsequence at the character level, the algorithm can discover the inherent connections between words, accurately identifying their similarity even with variations in form such as hyphens, spaces, and word order adjustments.

[0043] Furthermore, this application utilizes the character-level analysis capabilities of the Sequence Matcher algorithm to achieve intelligent filtering of cross-language vocabulary. By analyzing the character composition and structural patterns of strings, the algorithm can identify the characteristic differences between different languages, such as the distribution patterns of Chinese characters, English letters, and numerical symbols. When words with significantly different character sets appear in the expanded word list, their LCS length is usually short and their similarity ratio value is low. By filtering with a preset threshold, irrelevant minority language interference words can be automatically eliminated, maintaining the linguistic consistency and quality stability of the results.

[0044] Furthermore, the final expanded word recommendation list data is obtained, including: obtaining the similarity scores of expanded words in the fourth candidate data. and sort position index Based on similarity score and sort position index Calculate the weight of the expanded words. : Where α and β are coefficients. Let i be the similarity score for the i-th extended word. The index of the sorting position of the i-th expanded word; determine the highest weight. If the value is less than a preset threshold, clear the fourth candidate data and return an empty extended word recommendation list; otherwise, proceed to the next step. Use a linear transformation to correct the weights, obtaining the corrected weights. : ;in, and These represent the maximum and minimum weights of the expanded word sequence after descending order, respectively; and These represent the upper and lower limits of the target score range, respectively; the target score range represents the preset standardized score range; based on the adjusted weights... Sort the expanded word list in descending order to obtain the final expanded word recommendation list.

[0045] In particular, when all expanded words have the highest weight When all values ​​are below the preset threshold, the system determines that the retrieval quality is substandard, directly clears the candidate data and returns an empty recommendation list, avoiding pushing low-relevance extended words to users and preventing the system from making incorrect recommendations when the semantic matching degree is low.

[0046] Furthermore, the distinguishability of extended terms refers to the degree of difference and discriminability in similarity scores among different extended terms in the recommendation list. After multi-source retrieval fusion, the weight distribution of candidate extended terms often exhibits a concentrated characteristic, meaning that the score differences of most extended terms are small, resulting in a lack of clear priority distinction in the ranking results. In the hybrid retrieval fusion process, after weight calculation, the candidate data from the semantic retrieval module and the keyword retrieval module often exhibits an overly concentrated score distribution, such as the weights of most extended terms being concentrated in a narrow range of 0.6-0.8. This can be addressed through linear transformation. Remapping the original weights to a wider target range can effectively widen the score gap between expanded terms of different quality, creating a clear numerical distinction between high-quality and ordinary expanded terms. In literature retrieval scenarios, enhancing the discriminative power of expanded terms can help researchers quickly identify the most relevant research topics and keywords, avoiding ineffective filtering from a large number of expanded terms with similarity scores.

[0047] Another aspect of this application provides a document semantic search system based on elastic search, comprising: a first retrieval module that processes structured text data using an inverted index based on Apache Lucene and the BM25 algorithm; a second retrieval module that uses a Milvus vector database and processes vector data using a vector index structure; and a preprocessing module that preprocesses the input query text data, performs error correction by counting the number of characters, and uses Spacy... The `en_core_web_trf` model performs lexical and syntactic analysis, extracts noun phrases, and performs deduplication to generate a list of noun phrases. The vectorization module uses a `text2vec-base-multilingual` model trained based on the CoSENT method to vectorize the noun phrase list, converting each noun phrase into a high-dimensional semantic vector. The semantic retrieval module uses the semantic vector as the query vector and performs an approximate nearest neighbor search in the second retrieval module. It obtains candidate extended word data, similarity scores, and retrieval ranking positions through cosine similarity calculation and performs threshold filtering to obtain the first candidate data. The keyword retrieval module inputs the query text data into the first retrieval module and performs keyword matching retrieval using the BM25 algorithm. It establishes the correspondence between words and documents using an inverted index structure and calculates statistical scores based on term frequency and inverse document frequency, as well as the retrieval ranking position, as the second candidate data. The data fusion module merges the first and second candidate data and aggregates the merged candidate data according to logical operators in the query text data to obtain the third candidate data. The deduplication module uses a sequence... The Matcher algorithm calculates the string similarity between extended words in the third candidate data, sets a similarity threshold based on the length of the longest common subsequence, and performs deduplication to obtain the fourth candidate data. The weight allocation module assigns weights to the fourth candidate data based on position and similarity scores, expands the score range through linear transformation to enhance the distinguishability of extended words, and sorts them in descending order to obtain the final extended word recommendation list data.

[0048] Compared to existing technologies, the advantages of this application are:

[0049] Traditional Elasticsearch (ES) based on keyword matching cannot understand the deep semantics of queries. A single retrieval method struggles to simultaneously meet the needs of precise matching and semantic understanding, and ES inverted indexes are inefficient when processing high-dimensional vector data. This application constructs a dual-module hybrid retrieval architecture that separates traditional statistical keyword matching from deep learning-based semantic vector retrieval. A text2vec-base-multilingual model trained using the CoSENT method is employed to convert query text into high-dimensional semantic vectors. The Milvus vector database is used for approximate nearest neighbor search, achieving a technological leap from literal matching to semantic understanding. This enables the system to identify synonyms and near-synonyms and understand contextual semantic relationships.

[0050] Furthermore, this application employs a parallel dual-channel retrieval strategy, simultaneously running Apache Lucene's BM25 algorithm for precise keyword matching and Milvus's vector similarity calculation for semantic retrieval. Then, an intelligent fusion algorithm aggregates the two retrieval results based on logical operators, achieving a complementary advantage between precise matching and semantic understanding. This ensures both the high accuracy of traditional search and the high recall rate of semantic retrieval. Attached Figure Description

[0051] This application will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0052] Figure 1 This is an exemplary flowchart of a document semantic search method based on elastic search, according to some embodiments of this application;

[0053] Figure 2 This is an exemplary flowchart of ES retrieval according to some embodiments of this application;

[0054] Figure 3 This is an exemplary flowchart of Milvus semantic retrieval according to some embodiments of this application. Detailed Implementation

[0055] The methods and systems provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0056] Example 1

[0057] like Figure 1As shown, a hybrid retrieval architecture is constructed, comprising a first retrieval module and a second retrieval module. The first retrieval module uses Elasticsearch based on Apache Lucene, processing structured text data through an inverted index and the BM25 algorithm. The second retrieval module uses the Milvus vector database, processing vector data using a vector index structure. The input query text data is preprocessed to generate a list of noun phrases. The noun phrase list is vectorized using a text2vec-base-multilingual model trained based on the CoSENT method to obtain semantic vectors. These semantic vectors are used as query vectors, and an approximate nearest neighbor search is performed in the second retrieval module to obtain extended word data, similarity scores, and search ranking positions, serving as the first candidate data. The query text data is then input into the first retrieval module, where keyword matching is performed using the BM25 algorithm. Statistical scores based on term frequency and inverse document frequency, as well as the search ranking positions, are calculated, serving as the second candidate data. The first and second candidate data are merged, and the merged candidate data is aggregated based on logical operators in the query text data to obtain the third candidate data. A sequence is then used. The Matcher algorithm calculates the string similarity between extended words in the third candidate data, sets a similarity threshold based on the length of the longest common subsequence, and performs deduplication to obtain the fourth candidate data. The fourth candidate data is then weighted based on position and similarity scores, and the distinguishability of extended words is enhanced by expanding the score range, resulting in the final extended word recommendation list data.

[0058] like Figure 2 As shown, this solution is based on Elasticsearch (ES) and Milvus to build a high-efficiency and scalable document retrieval system, aiming to achieve rapid retrieval, accurate matching, and intelligent ranking of large-scale document data. By combining ES's distributed architecture, full-text search capabilities, and modern natural language processing (NLP) technology, the system not only supports traditional keyword retrieval but also introduces semantic retrieval and personalized recommendation functions to meet users' diverse needs for document retrieval.

[0059] Elasticsearch (ES) is built on Apache Lucene and uses inverted indexes and the BM25 algorithm to achieve efficient full-text search. The inverted index maps each word in a document to a list of documents containing that word, supporting rapid location of relevant documents. The BM25 algorithm calculates the relevance score between query keywords and documents by statistically analyzing term frequency (TF), inverse document frequency (IDF), and document length. The formula is: Formula explanation: D: Document. Q: Query, consisting of 1 to n queries. , term Term frequency (TF) in document D. The length (number of words) of document D. The average length of all documents in the document collection. b: Adjustable parameter b=0.75. , term The inverse document frequency (IDF) is calculated using the following formula: N: The total number of documents in the document collection. , containing terms The number of documents.

[0060] like Figure 3 As shown, in the semantic retrieval section, we chose Milvus to achieve accurate semantic search. The entire process includes the following core steps: input processing and language correction, semantic vector search, result merging and filtering, and weight allocation and output.

[0061] Automatic error correction based on word length: Utilizing pycorrector for text error correction, the function is activated when the input character count exceeds 5. It intelligently identifies and corrects spelling errors in text, helping users promptly identify potential problems. Error detection: Identifying spelling errors in text through dictionaries or language models. By comparing with a standard vocabulary, EnSpellCorrector can quickly find words not in the dictionary. Processing long text: Using the Spacy model for in-depth analysis of the input text, extracting key noun phrases, and decomposing long text into more easily understood semantic core units, thus providing efficient and accurate input for subsequent vectorization, semantic expansion, and retrieval. For text after automatic error correction, if its length is greater than 2, the Spacy en_core_web_trf model is applied for word segmentation, and noun phrases are extracted using doc.noun_chunks. Duplicates are then removed using a set structure, ultimately returning a list of phrases for subsequent vectorization and semantic expansion.

[0062] This project employs a text2vec-base-multilingual model trained using the CoSENT method. This model can convert millions of article keywords into vector form and store them in a vector database to support efficient semantic retrieval and semantic expansion word recommendation. Cosine similarity is used for vector comparison.

[0063] CoSENT calculates cosine similarity for sentence pair embeddings: ;in It is an embedding of two sentences.

[0064] CoSENT uses a ranking loss function based on cosine similarity. The formula for the loss function is as follows: , The set of all positive sample pairs The set of all negative sample pairs. The core of CoSENT is a new loss function that optimizes the cosine value.

[0065] Merging: Logical operators are used to process the query results. When the logical operator is the first operator (AND), the similarity scores of repeated extended words are accumulated to strengthen the intersection result; when the logical operator is the second operator (OR), the similarity scores of repeated extended words are averaged to balance the union result; when the logical operator is the third operator (NOT), the corresponding extended words are deleted from the candidate word set.

[0066] Filtering: Sequence Matcher is used to calculate string similarity, filtering out highly similar words to ensure the uniqueness and accuracy of the results. The similarity formula is: , where LCS is the longest common subsequence; a and b are both strings.

[0067] By combining the traditional Elasticsearch (ES) module with a semantic search module, a highly efficient and accurate document retrieval function was ultimately achieved. The traditional ES module is built on Apache Lucene and uses inverted indexes and the BM25 algorithm to achieve keyword retrieval, which can quickly locate documents that match the query keywords. However, its limitation lies in its inability to understand semantics, resulting in insufficient ability to process synonyms, near-synonyms, and contextual information.

[0068] To address the shortcomings of existing technologies in handling complex logical relationships, this project developed a semantic expanded word recommendation system based on semantic vector retrieval and weight allocation mechanisms. This system integrates the Milvus vector database, the text2vec tool, and natural language processing (NLP) techniques, aiming to enhance the ability to generate highly relevant expanded words under complex logical relationships. The following are some of the system's core features:

[0069] The system merges results from Elasticsearch (ES) and Milvus vector database searches and dynamically adjusts the result merging strategy: based on user-input logical conditions such as AND, OR, and NOT, the system can flexibly adjust the result merging method to improve the relevance of expanded keywords. This method overcomes the problem that traditional keyword expansion methods cannot effectively handle complex logical relationships.

[0070] Employing the Sequence Matcher algorithm, the system optimizes input and corrects spelling: By implementing spell correction, phrase recognition, and language restrictions, the system significantly improves the quality of input content and generates semantically complete and accurate candidate words. This approach particularly enhances the error tolerance of cross-language searches (e.g., between Chinese and English).

[0071] Efficient vector retrieval and filtering mechanism: Utilizing the vector similarity search function implemented by Milvus, combined with specific thresholds, filtering and removing duplicate words ensure the relevance and purity of the final results.

[0072] Intelligent weight allocation strategy: Obtain similarity scores for expanded words in candidate data. and sort position index Based on similarity score and sort position index Calculate the weight of the expanded words. : Where α and β are coefficients. Let i be the similarity score for the i-th extended word. The index of the sorting position of the i-th expanded word; determine the highest weight. If the value is less than a preset threshold, clear the fourth candidate data and return an empty extended word recommendation list; otherwise, proceed to the next step. Use a linear transformation to correct the weights, obtaining the corrected weights. : ;in, and These represent the maximum and minimum weights of the expanded word sequence after descending order, respectively; and These represent the upper and lower limits of the target score range, respectively; the target score range represents the preset standardized score range; based on the adjusted weights... Sort the expanded word list in descending order to obtain the final expanded word recommendation list.

[0073] This solution significantly improves the accuracy of semantic expansion and user experience across various retrieval scenarios, and is particularly suitable for searching and recommending various resources such as articles, books, journals, and preprints. Through this innovation, users can more easily find the information they truly need, while also bringing new ideas and technical means to the field of information retrieval.

[0074] The system first applies spell correction technology to correct input errors, then uses the en_core_web_trf model for word segmentation, and identifies key concepts by extracting noun phrases from doc.noun_chunks. Next, the text2vec tool is used to vectorize the text, and the Milvus vector database is used to achieve efficient related word search, thus completing the recommendation of expanded words.

[0075] The system specifically supports the processing of AND, OR, and NOT logical groupings. Through merging strategies and similarity deduplication (using the Sequence Matcher method), it effectively improves the relevance of expanded terms. In addition, the system implements language restrictions, weight allocation mechanisms, and duplicate word filtering. These measures together improve retrieval efficiency in long text processing and complex query scenarios, thereby significantly improving the user experience.

[0076] Overall, this intelligent semantic search system not only enhances the quality and diversity of expanded terms but also ensures the accuracy and efficiency of the retrieval process, making it an ideal choice for searching articles, books, journals, and other types of content. Through this innovative approach, users can more easily find the information they need and enjoy a smoother search experience.

[0077] Example 2

[0078] A document semantic search system based on elastic search includes: a first retrieval module that processes structured text data using an inverted index based on Apache Lucene and the BM25 algorithm; a second retrieval module that uses the Milvus vector database and processes vector data using a vector index structure; and a preprocessing module that preprocesses the input query text data, performs error correction by counting the number of characters, and uses Spacy... The `en_core_web_trf` model performs lexical and syntactic analysis, extracts noun phrases, and performs deduplication to generate a list of noun phrases. The vectorization module uses a `text2vec-base-multilingual` model trained based on the CoSENT method to vectorize the noun phrase list, converting each noun phrase into a high-dimensional semantic vector. The semantic retrieval module uses the semantic vector as the query vector and performs an approximate nearest neighbor search in the second retrieval module. It obtains candidate extended word data, similarity scores, and retrieval ranking positions through cosine similarity calculation and performs threshold filtering to obtain the first candidate data. The keyword retrieval module inputs the query text data into the first retrieval module and performs keyword matching retrieval using the BM25 algorithm. It establishes the correspondence between words and documents using an inverted index structure and calculates statistical scores based on term frequency and inverse document frequency, as well as the retrieval ranking position, as the second candidate data. The data fusion module merges the first and second candidate data and aggregates the merged candidate data according to logical operators in the query text data to obtain the third candidate data. The deduplication module uses a sequence... The Matcher algorithm calculates the string similarity between extended words in the third candidate data, sets a similarity threshold based on the length of the longest common subsequence, and performs deduplication to obtain the fourth candidate data. The weight allocation module assigns weights to the fourth candidate data based on position and similarity scores, expands the score range through linear transformation to enhance the distinguishability of extended words, and sorts them in descending order to obtain the final extended word recommendation list data.

[0079] The foregoing illustrative description of the present application and its embodiments is not restrictive and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. The accompanying drawings are only one embodiment of the present application, and the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present application, such designs should fall within the scope of protection of this application. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A document semantic search method based on elastic search, characterized in that, include: A hybrid retrieval architecture is constructed, comprising a first retrieval module and a second retrieval module. The first retrieval module uses Elasticsearch based on Apache Lucene to process structured text data through inverted indexes and the BM25 algorithm. The second retrieval module uses Milvus vector database to process vector data using vector index structures. The input query text data is preprocessed to generate a list of noun phrases; The text2vec-base-multilingual model trained based on the CoSENT method is used to vectorize the list of noun phrases to obtain semantic vectors; Using the semantic vector as the query vector, an approximate nearest neighbor search is performed in the second retrieval module to obtain extended word data, similarity scores, and retrieval ranking positions, which are then used as the first candidate data. Input the query text data into the first retrieval module, perform keyword matching retrieval using the BM25 algorithm, calculate the statistical score based on term frequency and inverse document frequency, and the retrieval ranking position, as the second candidate data; The first and second candidate data are merged, and the merged candidate data is aggregated according to the logical operators in the query text data to obtain the third candidate data. The SequenceMatcher algorithm is used to calculate the string similarity between extended words in the third candidate data. A similarity threshold is set based on the length of the longest common subsequence and deduplication is performed to obtain the fourth candidate data. The fourth candidate data is weighted based on position and similarity scores. The score range is expanded to enhance the distinguishability of the extended words, resulting in the final extended word recommendation list data. The third candidate data was obtained, including: Retrieve logical operators from the query text data. The logical operators include the first operator representing AND, the second operator representing OR, and the third operator representing NOT. The matching results from the first and second candidate data are merged, and the expanded words are extracted to form a candidate vocabulary set. Identify repeated expanded words in the candidate vocabulary set and aggregate them according to logical operators: When the logical operator is the first operator AND, the similarity scores of repeated extended words are accumulated to strengthen the intersection results; When the logical operator is the second operator "or", the similarity scores of repeated expanded words are averaged to balance the union results; When the logical operator is the third operator not, the corresponding extended word is removed from the candidate vocabulary set; The original similarity scores of the non-repeating expanded words in the first and second candidate data are retained and merged with the repeated expanded words after aggregation. Calculate the sort position index of the merged candidate data to generate a dataset containing expanded words, similarity scores, and sort position indices, which serves as the third candidate data.

2. The document semantic search method based on elastic search according to claim 1, characterized in that: Generate a list of noun phrases, including: Count the number of characters N in the query text data. When N is greater than a threshold... When the query text data is corrected, pycorrector, which is based on a statistical language model, is used to perform text correction on the query text data and output the corrected query text data; otherwise, the original query text data is used as the corrected query text data. Determine if the length of the corrected query text data is greater than the threshold. If yes, the Spacyen_core_web_trf model based on the Transformer architecture is used to perform lexical and syntactic analysis on the corrected query text data to generate document object data containing part-of-speech tags and grammatical structures; otherwise, the corrected query text data is used as document object data. Based on the document object data, extract the set of noun phrases using the doc.noun_chunks method built into the Spacy model; The set data structure is used to remove duplicates from the set of noun phrases, resulting in the final list of noun phrases.

3. The document semantic search method based on elastic search according to claim 1, characterized in that: The semantic vector is obtained, including: The text2vec-base-multilingual model is trained using the CoSENT method and the ranking loss function L based on cosine similarity. Using the trained text2vec-base-multilingual model, the list of noun phrases is semantically encoded, and each noun phrase is converted into a high-dimensional semantic vector; The expression for the sorting loss function L is as follows: Where i, j, k, l represent the sample indices in the sample pair. This represents the sentence embedding vector of the corresponding sample. Represents the set of all positive sample pairs. Represents the set of all negative sample pairs; This is a boundary parameter used to control the interval between positive and negative sample pairs; in, .

4. The document semantic search method based on elastic search according to claim 2, characterized in that: Obtain the first candidate data, including: Input the semantic vector as the query vector into the Milvus vector database; Perform an approximate nearest neighbor search based on cosine similarity in the Milvus vector database to calculate the similarity score between the query vector and the word vectors stored in the Milvus vector database. Obtain the top 1 candidate extended words with the highest similarity scores. The candidate extended word data includes the extended word text, similarity score, and sorting position index in the search results. The top 1 candidate extended word data is filtered by threshold. It is determined whether the similarity score of the first-ranked extended word is lower than the preset threshold. If it is, it is determined that there is no valid synonym extended word and empty first candidate data is returned; otherwise, the candidate extended word data that meets the threshold requirement is retained. The filtered candidate expanded word data, similarity score, and sort position index are combined into a triplet, which is used as the first candidate data.

5. The document semantic search method based on elastic search according to claim 2, characterized in that: Obtain the second candidate data, including: Input the query text data into the first search module based on Apache Lucene; By using an inverted index structure, each word in the query text data is mapped to a list of documents containing the corresponding word, thus establishing a correspondence between words and documents; The similarity score between query text data Q and document D is calculated based on the BM25 algorithm. ; Obtain the top two matching documents with the highest scores in the BM25 algorithm. The matching document results include document identifier, similarity score, and position index in the sorting. The top two matching literature results were used as the second candidate data.

6. The document semantic search method based on elastic search according to claim 5, characterized in that: Similarity score : Where Q represents the query text data, containing 1 to n query terms. ;D indicates a document; Indicates terms The term frequency TF in reference D; This indicates the word count of document D; This represents the average length of all documents in the document collection; b represents a parameter; The value of is in the range of 1.2 to 2; the value of b is in the range of 0.5 to 0.

85. Indicates terms Inverse document frequency (IDF): Where N represents the total number of documents in the document collection; Indicates included terms The number of documents.

7. The document semantic search method based on elastic search according to claim 2, characterized in that: The fourth candidate data was obtained, including: Obtain the extended word text from the third candidate data to form a list of extended words to be processed; The SequenceMatcher algorithm is used to calculate the string similarity ratio between any two words in the expanded word list: ;in, This represents the length of the longest common subsequence between strings a and b; and These represent the lengths of string a and string b, respectively. Determine if the string similarity ratio is greater than a preset threshold; if so, perform deduplication. The fourth candidate data is obtained by combining the deduplicated expanded words, their corresponding similarity scores, and the sorting position index.

8. The document semantic search method based on elastic search according to any one of claims 2 to 7, characterized in that: The final expanded keyword recommendation list data is obtained, including: Obtain the similarity score of the expanded words in the fourth candidate data. and sort position index ; Based on similarity score and sort position index Calculate the weight of the expanded words. : Where α and β are coefficients. Let i be the similarity score for the i-th extended word. This is the sorting position index of the i-th extended word; Determine the highest weight If the value is less than a preset threshold, clear the fourth candidate data and return an empty extended word recommendation list; otherwise, proceed to the next step. The weights are corrected using a linear transformation to obtain the corrected weights. : ;in, and These represent the maximum and minimum weights of the expanded word sequence after descending order, respectively; and These represent the upper and lower limits of the target score range, respectively; the target score range represents the preset standardized score range. Based on the corrected weights Sort the expanded word list in descending order to obtain the final expanded word recommendation list.

9. A document semantic search system based on elastic search, characterized in that, include: The first retrieval module processes structured text data using an inverted index based on Apache Lucene and the BM25 algorithm; The second retrieval module uses the Milvus vector database and processes vector data using a vector index structure. The preprocessing module preprocesses the input query text data, performs error correction by counting the number of characters, uses the Spacyen_core_web_trf model for lexical and syntactic analysis, extracts noun phrases and performs deduplication, and generates a list of noun phrases. The vectorization module uses a text2vec-base-multilingual model trained based on the CoSENT method to vectorize the list of noun phrases, converting each noun phrase into a high-dimensional semantic vector; The semantic retrieval module uses the semantic vector as the query vector and performs an approximate nearest neighbor search in the second retrieval module. It obtains candidate extended word data, similarity scores, and retrieval ranking positions through cosine similarity calculation, and performs threshold filtering to obtain the first candidate data. The keyword retrieval module inputs the query text data into the first retrieval module, performs keyword matching retrieval using the BM25 algorithm, establishes the correspondence between words and documents using the inverted index structure, calculates the statistical scores based on term frequency and inverse document frequency, and the retrieval ranking position, which serve as the second candidate data. The data fusion module merges the first and second candidate data, and aggregates the merged candidate data according to the logical operators in the query text data to obtain the third candidate data. The deduplication module uses the SequenceMatcher algorithm to calculate the string similarity between extended words in the third candidate data, sets a similarity threshold based on the length of the longest common subsequence, and performs deduplication to obtain the fourth candidate data. The weight allocation module assigns weights to the fourth candidate data based on position and similarity scores. It expands the score range through linear transformation to enhance the distinguishability of the extended words and sorts them in descending order to obtain the final extended word recommendation list data. The third candidate data obtained includes: Retrieve logical operators from the query text data. The logical operators include the first operator representing AND, the second operator representing OR, and the third operator representing NOT. The matching results from the first and second candidate data are merged, and the expanded words are extracted to form a candidate vocabulary set. Identify repeated expanded words in the candidate vocabulary set and aggregate them according to logical operators: When the logical operator is the first operator AND, the similarity scores of repeated extended words are accumulated to strengthen the intersection results; When the logical operator is the second operator "or", the similarity scores of repeated expanded words are averaged to balance the union results; When the logical operator is the third operator not, the corresponding extended word is removed from the candidate vocabulary set; The original similarity scores of the non-repeating expanded words in the first and second candidate data are retained and merged with the repeated expanded words after aggregation. Calculate the sort position index of the merged candidate data to generate a dataset containing expanded words, similarity scores, and sort position indices, which serves as the third candidate data.

Images