A semantic retrieval method, system, device, and storage medium based on an inverted index.
By constructing a semantic inverted index, keywords are converted into semantic vectors and clustered, solving the problem that traditional inverted indexes cannot understand semantics, achieving efficient semantic retrieval, improving recall and precision, and meeting the real-time requirements of online services.
Patent Information
- Application Number
- CN202611499864.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-24
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, traditional inverted indexes cannot understand semantics, resulting in low recall rates, while pure semantic vector retrieval has high computational overhead and is difficult to meet the real-time requirements of online services.
By converting keywords into semantic vectors and clustering them, a semantic inverted index is constructed with semantic group identifiers as index terms. Combining traditional inverted indexes and semantic understanding, efficient semantic retrieval is achieved.
It improves the recall and precision of retrieval, meets the real-time requirements of online services, and can process structured, semi-structured, and unstructured data.
Smart Images

Figure CN123115295A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information retrieval technology, specifically providing a semantic retrieval method, system, device, and storage medium based on an inverted index. Background Technology
[0002] Information retrieval is one of the core problems in computer science. Currently, mainstream retrieval technologies are mainly divided into two categories: Searching based on traditional inverted indexes: This is the cornerstone of modern search engines. Its core idea is to establish a mapping from "keywords" to a "list of documents containing that keyword." For example, the index entry "apple" points to all documents containing the word "apple." The advantage of this method is its extremely fast retrieval speed and relatively low construction and maintenance costs. However, its fatal flaw is its inability to understand semantics; it can only perform exact literal matching. For example, if a user searches for "notebook," the system cannot return documents containing "laptop" or "handheld computer"—a phenomenon known as the "lexical mismatch problem."
[0003] Semantic Vector-Based Retrieval: With the development of deep learning, semantic vector technology, represented by models such as Word2Vec, GloVe, and BERT, has been widely applied to retrieval. Its core idea is to map text (words, sentences, documents) into vectors in a high-dimensional space, and measure semantic relevance by calculating cosine similarity between vectors. The advantage of this method is its ability to deeply understand the inherent semantics of text and effectively solve the word mismatch problem. However, its disadvantages are also quite obvious: First, the computational cost of real-time vector similarity calculation for massive amounts of documents is enormous, and the response time is difficult to meet the requirements of online search; second, pure vector retrieval cannot efficiently utilize the structured information in the text (such as product prices, categories, authors, etc.).
[0004] To combine the advantages of both, existing technologies have attempted hybrid approaches, such as "first using an inverted index for initial screening, then using a vector model for reordering." However, this is essentially still two separate steps. The initial screening stage may have already lost a large number of relevant documents due to word mismatches, resulting in limited final recall.
[0005] Therefore, designing a retrieval method that combines the efficiency of traditional inverted indexes with the accuracy of semantic understanding is a pressing technical challenge in the field of information retrieval. Summary of the Invention
[0006] The technical objective of this invention is to provide a semantic retrieval method, system, device, and storage medium based on inverted indexes, in order to solve the technical problems of low recall rate caused by the inability of traditional inverted indexes to understand semantics, and high computational overhead and difficulty in meeting the real-time requirements of online services for pure semantic vector retrieval.
[0007] The technical solution adopted by this invention to solve its technical problem is: A semantic retrieval method based on an inverted index includes: S1. Data Acquisition and Preprocessing: Acquire the data sources to be retrieved, including structured data, semi-structured data, or unstructured data; clean and standardize the format of the data sources, extract the text content to be indexed, and assign a unique document identifier to each data source; S2. Keyword Extraction and Vectorization: Extract keywords from the text content of each document and use a pre-trained language model to convert each keyword into a corresponding semantic vector; S3. Semantic Keyword Grouping: Using the keywords extracted from all documents and their corresponding semantic vectors as input, a clustering algorithm is used to group semantically similar keywords into the same semantic cluster. A unique semantic group identifier is assigned to each semantic cluster, thereby establishing a mapping relationship between semantic group identifiers and keyword sets. S4. Semantic Inverted Index Construction: Traverse each document, extract keywords from the document to obtain a keyword list, find the semantic group identifier to which each keyword belongs in the list, add the document identifier of the current document to the inverted list of the corresponding semantic group identifier, and finally construct a semantic inverted index with semantic group identifier as the index item. S5. Query processing: Receive user queries, perform the same processing as S1 and S2 on the user queries, extract query keywords and convert them into query semantic vectors, and then determine the query semantic group identifier corresponding to the query keywords according to the mapping relationship established in S3. S6. Retrieval and Sorting: Search the semantic inverted index according to the query semantic group identifier to obtain one or more candidate document identifier lists, merge and sort the lists to generate the final retrieval results.
[0008] More preferably, the textualization of structured or semi-structured data in S1 includes: extracting data fields and concatenating field names and field values into a text string.
[0009] Preferably, the keyword extraction in S2 uses the TF-IDF algorithm or the TextRank algorithm; the pre-trained language model is the BERT model or the Word2Vec model.
[0010] Preferably, the clustering algorithm in S3 is the K-Means algorithm or the DBSCAN algorithm.
[0011] More preferably, the structure of the semantic inverted index constructed in S4 is: {semantic group identifier G_i: [document identifier Doc_1, document identifier Doc_2, ...]}, where the index item is a semantic group identifier rather than a single keyword.
[0012] More preferably, the method of merging the candidate document identifier list in S5 includes taking the union or taking the intersection.
[0013] Furthermore, the sorting step in S6 further includes: combining the BM25 algorithm based on word frequency and the cosine similarity algorithm based on vector similarity to perform a comprehensive scoring and sorting of candidate documents.
[0014] A semantic retrieval system based on an inverted index, comprising: The data acquisition and preprocessing module is used to acquire data sources to be retrieved, including structured data, semi-structured data, or unstructured data; clean and format the data sources, extract the text content to be indexed, and assign a unique document identifier to each data source; The keyword extraction and vectorization module is used to extract keywords from the text content of each document and convert each keyword into a corresponding semantic vector using a pre-trained language model. The semantic keyword grouping module takes the keywords extracted from all documents and their corresponding semantic vectors as input, and uses a clustering algorithm to group semantically similar keywords into the same semantic cluster. It assigns a unique semantic group identifier to each semantic cluster, thereby establishing a mapping relationship between semantic group identifiers and keyword sets. The semantic inverted index building module is used to traverse each document, extract keywords from the document to obtain a keyword list, find the semantic group identifier to which each keyword in the list belongs, add the document identifier of the current document to the inverted list of the corresponding semantic group identifier, and finally build a semantic inverted index with semantic group identifier as the index item. The query processing module is used to receive user queries, perform the same processing as the data acquisition and preprocessing module and the keyword extraction and vectorization module on the user queries, extract query keywords and convert them into query semantic vectors, and then determine the query semantic group identifier corresponding to the query keywords according to the mapping relationship established by the semantic keyword grouping module. The retrieval and sorting module is used to search in the semantic inverted index according to the query semantic group identifier, obtain one or more candidate document identifier lists, merge and sort the lists, and generate the final retrieval results.
[0015] An electronic device includes: at least one processor; and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the semantic inverted index-based retrieval method.
[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semantic inverted index-based retrieval method.
[0017] Compared with existing technologies, the semantic retrieval method, system, device, and storage medium based on inverted index of the present invention have the following outstanding advantages: This invention integrates semantic understanding capabilities into the index structure, which retains the efficiency of traditional inverted indexes while achieving the accuracy of semantic retrieval, significantly improving the recall and precision of retrieval, and effectively processing structured, semi-structured and unstructured data to meet the real-time requirements of online services. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart illustrating a semantic retrieval method based on an inverted index. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] The following is a preferred embodiment: Example
[0022] This embodiment of a semantic retrieval method based on an inverted index includes: S1. Data Acquisition and Preprocessing: Collect the data sources to be retrieved, including structured, semi-structured, or unstructured data; clean and format the data sources, extract the text content to be indexed, and assign a unique document identifier to each data source.
[0023] Textification of structured or semi-structured data includes extracting data fields and concatenating field names and values into a text string.
[0024] S2. Keyword extraction and vectorization: Keyword extraction is performed on the text content of each document, and each keyword is converted into a corresponding semantic vector using a pre-trained language model; In this embodiment, the keyword extraction uses the TF-IDF algorithm, and the pre-trained language model is the BERT model.
[0025] S3. Semantic keyword grouping: The keywords extracted from all documents and their corresponding semantic vectors are used as input. A clustering algorithm is used to group semantically similar keywords into the same semantic cluster. A unique semantic group identifier is assigned to each semantic cluster, thereby establishing a mapping relationship between the semantic group identifier and the keyword set.
[0026] The K-Means algorithm was chosen for clustering.
[0027] S4. Semantic Inverted Index Construction: Each document is traversed, keywords are extracted to obtain a keyword list, the semantic group identifier of each keyword in the list is found, the document identifier of the current document is added to the inverted list of the corresponding semantic group identifier, and finally a semantic inverted index is constructed with semantic group identifiers as index items.
[0028] The structure of the constructed semantic inverted index is: {semantic group identifier G_i: [document identifier Doc_1, document identifier Doc_2, ...]}, where the index item is a semantic group identifier rather than a single keyword.
[0029] The specific method is as follows: Iterate through each document: a. Extract keywords from the document to obtain its keyword list.
[0030] b. For each keyword in the list, find its semantic group ID.
[0031] c. Add the ID of the current document (Doc_ID) to the inverted list corresponding to the semantic group ID.
[0032] The final index structure is: Semantic Inverted Index = { G_001: [Doc_1, Doc_5, ...], G_002: [Doc_2, Doc_3, ...], ...}. Unlike traditional indexes, index entries are no longer single words, but groups representing semantic concepts.
[0033] S5. Query Processing: Receive user queries, perform the same processing as S1 and S2 on the user queries, extract query keywords and convert them into query semantic vectors, and then determine the query semantic group identifier corresponding to the query keywords according to the mapping relationship established in S3.
[0034] In this embodiment, the candidate document identifier list is merged by taking the intersection.
[0035] S6. Search and sorting: The semantic inverted index is searched based on the query semantic group identifier to obtain one or more candidate document identifier lists. The lists are then merged and sorted to generate the final search results.
[0036] The ranking step further includes: combining the BM25 algorithm based on word frequency and the cosine similarity algorithm based on vector similarity to perform a comprehensive scoring and ranking of candidate documents.
[0037] Query processing and retrieval: a. Receive user queries.
[0038] b. Perform the same operations on the query as document preprocessing: keyword extraction and vectorization.
[0039] c. Determine the semantic group ID to which each keyword in the query belongs.
[0040] d. Based on these semantic group IDs, look up the corresponding list of document IDs in the semantic inverted index.
[0041] e. Merge multiple lists (e.g., take the union or intersection) to obtain a preliminary set of candidate documents.
[0042] f. (Optional) Perform fine-grained ranking of the candidate document set, which can be done by combining the traditional BM25 algorithm with the semantic similarity score between the document and the query vector to obtain the final ranking result and return it. Example
[0043] Taking an e-commerce platform containing product information as an example, the data source includes product titles (unstructured text), product details (JSON format, semi-structured), and product specifications (database table, structured).
[0044] S1. Data acquisition and preprocessing; Suppose we have the following data: Doc1 (unstructured): "2023 high-performance gaming laptops equipped with the latest graphics cards." Doc2 (semi-structured JSON): {"name": "Business ultrabook", "category": "Computer", "desc": "Laptop suitable for office use"} Doc3 (Structured Table): ID: 003, Name: "Apple iPhone", Type: "Smartphone" After preprocessing, a uniform text stream is obtained: Doc1_text: "2023 High-Performance Gaming Laptops Equipped with the Latest Graphics Cards" Doc2_text: "Business thin and light laptops suitable for office use" Doc3_text: "Apple iPhone Smartphone" S2. Keyword extraction and vectorization; Keywords were extracted using TF-IDF (for simplicity, we will directly look at the word segmentation results here), and each word was converted into a 768-dimensional vector using the BERT model.
[0045] Vector ("notebook"), Vector ("computer"), Vector ("thin laptop"), … Vector("phone"), Vector("smartphone"), … S3. Semantic keyword grouping; Collect the keyword vectors ("notebook", "computer", "thin laptop", "phone", "smartphone", etc.) from all documents and cluster them using the K-Means algorithm. Let's assume we set K=2.
[0046] Clustering result 1 (representing the concept of "computer"): {"notebook", "computer", "thin laptop", "laptop"...}. The system assigns G_001 to this cluster.
[0047] Clustering result 2 (representing the concept of "mobile phone"): {"mobile phone", "smartphone", "mobile phone"...}. The system assigns G_002 to this cluster.
[0048] Other words may also form other groups, such as "graphics card" in G_005 (hardware components).
[0049] S4. Construct a semantic inverted index; Traverse each document and build an index: Doc1: The keyword "notebook" belongs to G_001, and "graphics card" belongs to G_005. Index update: G_001:[Doc1], G_005: [Doc1].
[0050] Doc2: Keywords “computer”, “thin and light laptop”, and “notebook” all belong to G_001. Index update: G_001:[Doc1, Doc2].
[0051] Doc3: The keywords "mobile phone" and "smartphone" both belong to G_002. Index update: G_002: [Doc3].
[0052] Final Index: { G_001: [Doc1, Doc2], G_002: [Doc3], G_005: [Doc1], ...} S5. Query processing and retrieval; User enters query: "laptop for office use" Query processing: Extract keywords "office" and "laptop".
[0053] Vectorization and Grouping: "Office use" may be classified under G_010 (purpose).
[0054] "laptop" was determined to belong to G_001 through vector and clustering.
[0055] Index lookup: The system searches for G_001 and G_010.
[0056] The document list corresponding to G_001 is [Doc1, Doc2].
[0057] G_010 is assumed to correspond to [Doc2, Doc4].
[0058] Merging and sorting: Take the intersection to get the candidate set [Doc2]. Or take the union [Doc1, Doc2, Doc4] and then sort.
[0059] Results returned: Doc2 (“business ultrabook”) was returned as the most relevant result. Even though Doc2 does not contain the word “laptop”, it was successfully recalled because its core keywords “notebook” and “computer” are in the same semantic group as “laptop”.
[0060] This embodiment demonstrates that the present invention effectively overcomes the barriers between Chinese and English vocabulary, understands the association between "notebook" and "computer", achieves accurate semantic recall, and the entire retrieval process is still based on efficient index lookup. Example
[0061] This embodiment of a semantic retrieval system based on an inverted index includes: The data acquisition and preprocessing module is used to acquire data sources to be retrieved, including structured data, semi-structured data, or unstructured data; clean and format the data sources, extract the text content to be indexed, and assign a unique document identifier to each data source. The keyword extraction and vectorization module is used to extract keywords from the text content of each document and convert each keyword into a corresponding semantic vector using a pre-trained language model. The semantic keyword grouping module takes the keywords extracted from all documents and their corresponding semantic vectors as input, and uses a clustering algorithm to group semantically similar keywords into the same semantic cluster. It assigns a unique semantic group identifier to each semantic cluster, thereby establishing a mapping relationship between semantic group identifiers and keyword sets. The semantic inverted index building module is used to traverse each document, extract keywords from the document to obtain a keyword list, find the semantic group identifier to which each keyword in the list belongs, add the document identifier of the current document to the inverted list of the corresponding semantic group identifier, and finally build a semantic inverted index with semantic group identifier as the index item. The query processing module is used to receive user queries, perform the same processing as the data acquisition and preprocessing module and the keyword extraction and vectorization module on the user queries, extract query keywords and convert them into query semantic vectors, and then determine the query semantic group identifier corresponding to the query keywords according to the mapping relationship established by the semantic keyword grouping module. The retrieval and sorting module is used to search in the semantic inverted index based on the query semantic group identifier, obtain one or more candidate document identifier lists, merge and sort the lists, and generate the final retrieval results. Example
[0062] An electronic device includes: a processor; and a memory communicatively connected to at least one processor; The memory stores instructions that can be executed by a processor, which enables at least one processor to perform a retrieval method based on a semantic inverted index.
[0063] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.
[0064] Memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices. Example
[0065] This embodiment provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to execute a retrieval method based on a semantic inverted index as described in any embodiment of the present invention.
[0066] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.
[0067] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0068] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network. Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0069] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A semantic retrieval method based on an inverted index, characterized in that, include: S1. Data Acquisition and Preprocessing: Acquire the data source to be retrieved, including structured data, semi-structured data, or unstructured data; The data sources are cleaned and formatted, the text content to be indexed is extracted, and a unique document identifier is assigned to each data source. S2. Keyword Extraction and Vectorization: Extract keywords from the text content of each document and use a pre-trained language model to convert each keyword into a corresponding semantic vector; S3. Semantic Keyword Grouping: Using the keywords extracted from all documents and their corresponding semantic vectors as input, a clustering algorithm is used to group semantically similar keywords into the same semantic cluster. A unique semantic group identifier is assigned to each semantic cluster, thereby establishing a mapping relationship between semantic group identifiers and keyword sets. S4. Semantic Inverted Index Construction: Traverse each document, extract keywords from the document to obtain a keyword list, find the semantic group identifier to which each keyword belongs in the list, add the document identifier of the current document to the inverted list of the corresponding semantic group identifier, and finally construct a semantic inverted index with semantic group identifier as the index item. S5. Query processing: Receive user queries, perform the same processing as S1 and S2 on the user queries, extract query keywords and convert them into query semantic vectors, and then determine the query semantic group identifier corresponding to the query keywords according to the mapping relationship established in S3. S6. Retrieval and Sorting: Search the semantic inverted index according to the query semantic group identifier to obtain one or more candidate document identifier lists, merge and sort the lists to generate the final retrieval results.
2. The semantic retrieval method based on an inverted index according to claim 1, characterized in that, The text processing of structured or semi-structured data in S1 includes: extracting data fields and concatenating field names and field values into a text string.
3. The semantic retrieval method based on an inverted index according to claim 1, characterized in that, The keyword extraction in S2 uses the TF-IDF algorithm or the TextRank algorithm; the pre-trained language model is the BERT model or the Word2Vec model.
4. The semantic retrieval method based on an inverted index according to claim 1, characterized in that, The clustering algorithm in S3 is either the K-Means algorithm or the DBSCAN algorithm.
5. A semantic retrieval method based on an inverted index according to claim 1, characterized in that, The structure of the semantic inverted index constructed in S4 is: {semantic group identifier G_i: [document identifier Doc_1, document identifier Doc_2,...]}, where the index item is a semantic group identifier rather than a single keyword.
6. The semantic retrieval method based on an inverted index according to claim 1, characterized in that, The method for merging the candidate document identifier list in S5 includes taking the union or taking the intersection.
7. The semantic retrieval method based on an inverted index according to claim 1, characterized in that, The sorting step in S6 further includes: combining the BM25 algorithm based on word frequency and the cosine similarity algorithm based on vector similarity to perform a comprehensive scoring and sorting of candidate documents.
8. A semantic retrieval system based on an inverted index, characterized in that, include: The data acquisition and preprocessing module is used to acquire data sources to be retrieved, including structured data, semi-structured data, or unstructured data. The data sources are cleaned and formatted, the text content to be indexed is extracted, and a unique document identifier is assigned to each data source. The keyword extraction and vectorization module is used to extract keywords from the text content of each document and convert each keyword into a corresponding semantic vector using a pre-trained language model. The semantic keyword grouping module takes the keywords extracted from all documents and their corresponding semantic vectors as input, and uses a clustering algorithm to group semantically similar keywords into the same semantic cluster. It assigns a unique semantic group identifier to each semantic cluster, thereby establishing a mapping relationship between semantic group identifiers and keyword sets. The semantic inverted index building module is used to traverse each document, extract keywords from the document to obtain a keyword list, find the semantic group identifier to which each keyword in the list belongs, add the document identifier of the current document to the inverted list of the corresponding semantic group identifier, and finally build a semantic inverted index with semantic group identifier as the index item. The query processing module is used to receive user queries, perform the same processing as the data acquisition and preprocessing module and the keyword extraction and vectorization module on the user queries, extract query keywords and convert them into query semantic vectors, and then determine the query semantic group identifier corresponding to the query keywords according to the mapping relationship established by the semantic keyword grouping module. The retrieval and sorting module is used to search in the semantic inverted index according to the query semantic group identifier, obtain one or more candidate document identifier lists, merge and sort the lists, and generate the final retrieval results.
9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the semantic inverted index-based retrieval method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the retrieval method based on a semantic inverted index as described in any one of claims 1 to 7.
