A document retrieval method that supports batch searching

Abstract

This invention discloses a literature retrieval method supporting batch retrieval, comprising: using Elasticsearch as the search engine, converting literature data into an ES index structure and importing relevant literature data, generating an index, and building an ES library; dividing each index into multiple primary shards, and distributing literature data to different primary shards through hash routing; each primary shard having at least one replica; constructing a tree-like nested Boolean query according to the retrieval target, using parallel queries during the query, and querying only the literature ID, storing the retrieved literature ID in a Redis Bitmap; when the user views the results in pages, retrieving the corresponding page literature ID from the Redis Bitmap, and reading the corresponding literature data from the ES library based on the literature ID.

Description

Technical Field

[0001] This invention relates to the field of document retrieval technology, and more specifically to a document retrieval method that supports batch retrieval. Background Technology

[0002] In fields such as academic research, intellectual property management, and corporate technological innovation, users often need to conduct batch literature searches based on multiple keywords, patent numbers, author names, and other information in order to quickly obtain technological trends, research results, or competitive intelligence in related fields.

[0003] Traditional document retrieval systems typically store document information in relational databases. When processing batch retrieval requests, they usually employ a loop-based, single-record search approach, resulting in low retrieval efficiency and high resource consumption. Furthermore, due to the lack of intelligent analysis and optimization for batch retrieval requests, search results often suffer from duplication and low relevance, failing to meet users' needs for accurate and efficient retrieval.

[0004] Elasticsearch, a distributed search engine based on Lucene, boasts powerful full-text search, distributed computing, and Boolean search capabilities, and has been widely applied in various information retrieval scenarios. However, most existing ES-based retrieval systems focus on processing single search requests, lacking support for batch searches and standardized processing procedures, intelligent search strategy generation, and result optimization mechanisms for batch search requests. Therefore, how to build an efficient and accurate document retrieval method and system that supports batch searches based on ES's Boolean search mechanism has become an urgent technical problem to be solved.

[0005] In existing technologies, when the data volume reaches hundreds of millions, the two-dimensional table structure of relational databases leads to excessively large single table sizes, deeper B+ tree index levels, and a surge in disk I / O overhead. Documentary data typically contains long texts (such as abstracts and full texts). If stored in a relational database, this not only consumes a large amount of storage space but also reduces the memory cache hit rate. Each retrieval may require frequent disk reads and writes, degrading response time from milliseconds to seconds or even longer. Furthermore, traditional LIMIT offset pagination requires scanning and discarding a large amount of preceding data during deep pagination (such as the 1 millionth page), resulting in extremely low efficiency.

[0006] Relational databases do not support multiple search matching methods, such as exact matching, fuzzy matching, range matching, multi-value matching, and word segmentation matching. The core requirement for document retrieval is often full-text search, which involves word segmentation, fuzzy matching, and relevance ranking of unstructured text. While relational databases support LIKE or basic full-text indexing, their performance is far inferior to dedicated search engines (such as Elasticsearch) when handling complex word segmentation, synonym expansion, and relevance scoring of hundreds of millions of data points. Relational databases excel at exact matching and transaction processing, but struggle to efficiently handle common document-related tasks such as fuzzy search, semantic association, and multi-dimensional aggregation analysis.

[0007] Slow retrieval speed: Relational databases mainly rely on vertical scaling (upgrading single-machine hardware). When facing hundreds of millions of data points, hardware costs rise exponentially and have a physical limit. Although database sharding and table partitioning can alleviate the pressure, this greatly increases the development complexity of the application layer, and cross-shard join queries have extremely poor performance, or may even be impossible to execute.

[0008] While relational databases do not completely lack support for batch fuzzy searches, they exhibit significant shortcomings in practical applications. They primarily rely on LIKE or REGEXP for fuzzy matching. When combined with batch conditions (such as multiple keywords), multiple fuzzy conditions must be concatenated using OR, which limits the length of the concatenated SQL and drastically reduces query efficiency, easily leading to full table scans. Even with indexes, leading wildcards (such as %keyword%) can render the index ineffective, resulting in uncontrollable performance. Although single-field fuzzy queries can be optimized using full-text indexes (such as MySQL's Full-Text), support for batch fuzzy searches involving multiple fields and keywords is weak, lacking efficient word segmentation, relevance ranking, and distributed processing capabilities.

[0009] Batch retrieval results suffer from high deduplication and sorting challenges: Hash indexes face efficiency bottlenecks when handling massive amounts of data. As the data volume increases, the cost of building and maintaining hash tables rises sharply, and query performance deteriorates due to increased hash collisions. While Bloom filters have a smaller space footprint, their inherent false positive rate is particularly prominent in document retrieval, potentially mislabeling non-duplicate documents as duplicates, leading to missed results. As for sorting, traditional sorting mechanisms suffer from performance degradation due to the need to load large amounts of data into memory for comparison.

[0010] Storing search results: When the literature search result set is large, such as hundreds of thousands of records in a single search, the database needs to read a large amount of data from disk into memory for processing. This may lead to memory overflow or excessive creation of temporary files, thus affecting system stability. Furthermore, literature data often contains long text content; when the database saves the search results as temporary tables, the size of each record is large, further exacerbating storage pressure. Even with index optimization and pagination mechanisms, when faced with complex queries involving massive amounts of data, the database may still be unable to efficiently return a complete result set due to I / O bottlenecks and lock contention. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing technologies by providing a document retrieval method that supports batch searching.

[0012] The objective of this invention is achieved through the following technical solution: a document retrieval method supporting batch retrieval, comprising: Using Elasticsearch as the search engine, the literature data is transformed into an ES index structure and relevant literature data is imported to generate an index and build an ES library; Each index is divided into multiple primary shards, and the literature data is distributed to different primary shards through hash routing; each primary shard has at least one replica (replica shard). Based on the search target, construct a tree-structured nested Boolean query, use parallel querying, and only query the document ID. Store the retrieved document IDs in a Redis Bitmap. When a user views the results in pages, the corresponding page document ID is retrieved from the Redis Bitmap, and the corresponding document data is read from the Elasticsearch database based on the document ID.

[0013] The query uses parallel querying and only queries the document ID, specifically: The Elasticsearch library is divided into several data ranges, and the document IDs are queried from each of the several data ranges according to the constructed tree-like nested Boolean query.

[0014] Furthermore, the literature data is transformed into an Elasticsearch index structure, including: Set the Title and Abstract text fields in the literature data to text type with keyword type subfields, and configure the ik_max_word word segmenter to meet both precise and fuzzy word segmentation matching methods. Set the Identifier and DOI text fields in the literature data to keyword type; The PublicationYear and Pageinfo numeric fields are set to number type to satisfy the matching retrieval of range values.

[0015] Furthermore, it also includes enabling a caching mechanism to speed up repeated queries.

[0016] Furthermore, based on the search objective, several search criteria are generated and filtered. Based on the filtered search criteria, a tree-like nested Boolean query is constructed.

[0017] Furthermore, the search criteria are filtered and selected, specifically: if the number of keywords in a single search criterion is less than a preset threshold, then the search criterion is filtered out.

[0018] Furthermore, the tree-structured nested Boolean query consists of several levels of Boolean queries, with the last level consisting of several search conditions (the search conditions are composed of ES search statements); the level is determined by the total number of ES Boolean query conditions limit and the number of search conditions.

[0019] Furthermore, the ES database is divided into several data intervals. Specifically, the maximum number of parallel queries that an ES node can support is multiplied by the expected total number of query batches to obtain the total number of data intervals. Then, the entire data in the ES database is divided equally according to the total number of data intervals. It also includes setting up a tree-like multi-level organization for Redis Bitmap, specifically: the results of every n data ranges are fed into the first-level Redis Bitmap, resulting in several Redis Bitmaps; the results of the n Redis Bitmaps in the first level are fed into the second-level Redis Bitmap, until the last level obtains a final result Redis Bitmap (the number of Redis Bitmaps in the penultimate level can be greater than 1 or less than or equal to n).

[0020] The present invention also provides a document retrieval device that supports batch retrieval, including one or more processors, for implementing the above-described document retrieval device that supports batch retrieval.

[0021] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described document retrieval method supporting batch retrieval.

[0022] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described document retrieval method supporting batch retrieval.

[0023] Compared with existing technologies, the beneficial effects of this invention are as follows: 1. Storage of massive amounts of literature data: Elasticsearch is based on a distributed architecture design, which can split and store massive amounts of data across multiple nodes in the cluster. The storage limit of a single node is only limited by the local hardware, and the entire cluster can easily support terabyte-level data. It adopts inverted index compression technology, which can significantly reduce the space occupied by data storage, while supporting tiered storage of hot and cold data, migrating infrequently accessed historical data to low-cost storage media, thus balancing storage capacity and cost efficiency.

[0024] 2. Multiple Matching Methods: Elasticsearch offers rich query DSL support, including keyword matching, fuzzy matching, phrase matching, and regular expression matching. Full-text fuzzy search is achieved through `match`, exact value matching through `term`, wildcard queries support wildcard queries, and combined with `bool` queries, multiple conditions can be matched to meet complex document retrieval needs.

[0025] 3. Improved retrieval speed: Based on inverted index technology, Elasticsearch establishes a mapping relationship between document content and keywords, directly locating the document corresponding to the keyword during retrieval, avoiding full table scans. In its distributed architecture, query requests are distributed in parallel to multiple nodes, combined with Lucene's underlying efficient search algorithms, achieving millisecond-level response times for terabyte-scale data.

[0026] 4. Supports Batch Search: Elasticsearch's boolean query, with its flexible logical combination capabilities, has become a core tool for meeting the needs of single-batch queries. Through the four clauses must, should, filter, and must_not, it allows for the free combination of various basic queries such as term, match, and range according to "AND / OR / NOT" logic. This enables precise filtering such as "title contains keywords and publication time is within a specified range," and also allows setting weights for different matching conditions through the should clause, prioritizing the display of highly relevant results. Furthermore, boolean queries support nested usage, allowing for the construction of multi-level complex logic to adapt to multi-dimensional filtering needs in batch scenarios. This combination method enables a single query to handle multiple dimensions of search requirements simultaneously, such as "must contain keywords A and B, can contain C or D, exclude type E documents, limited to the 2020-2024 range," achieving complex yet precise batch document retrieval.

[0027] 5. Parallel execution of pagination result retrieval: Elasticsearch's `search_after` function uses the sort value as a cursor to achieve efficient and repeatable deep pagination. The first query requires no special parameters, only specifying the sorting rule and page size. Subsequent requests use the sort value of the last document on the previous page (such as a timestamp and `_id`) as the `search_after` parameter to directly locate the starting point of the next page. This mechanism avoids the performance bottleneck of deep pagination using `from + size`, as each shard only needs to read data from after the specified sort value, without skipping a large number of records, significantly improving query efficiency. Because each query is a real-time retrieval, data changes are reflected immediately, supporting stable pagination under large offsets, making it suitable for massive data scenarios such as logs, documents, and orders.

[0028] However, `search_after` retrieves a certain number of search results starting from a given offset, guaranteeing no duplicates and sequential results. Its core drawback is its unidirectional cursor mechanism; it must iterate through each page sequentially, using the last `sort` value of the previous page as the starting offset for the next page. It cannot directly jump to page 500; it must retrieve the `sort` values ​​of the first 499 pages sequentially, starting from page 1.

[0029] This invention pre-divides the entire literature database into data intervals. After division, each data interval is assigned a fixed starting document ID, which is used as the `search_after` cursor. During paginated queries, queries are initiated in parallel for each data interval. The query results, i.e., document IDs, are finally aggregated and saved using a multi-level Redis Bitmap.

[0030] This method breaks the dependency between pages in paginated queries, predetermines the starting offset of each page, and obtains the final result set through efficient deduplication and sorting methods, thereby achieving parallel execution of paginated retrieval.

[0031] 6. Search results can be viewed repeatedly: Redis Bitmap maps document IDs to binary bits, allowing for efficient storage of hundreds of millions of entries per key (100 million IDs occupy only 12.5 MB). A single Bitmap stores all document IDs from a single search request. When performing deep pagination queries using Elasticsearch's `search_after` method, only document IDs within that page are retrieved and stored in the Bitmap, further accelerating Elasticsearch's search speed and solving the problem of storing large result sets. Subsequent result viewing extracts the paged document IDs and retrieves document details from Elasticsearch in real-time using these IDs. This not only ensures access to the latest document details in real time but also significantly improves the user experience by providing extremely fast results retrieval based on page IDs, typically within seconds. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of a document retrieval method supporting batch retrieval, provided as an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of a two-level nested Boolean query structure provided in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of data interval division provided in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of a document retrieval device that supports batch retrieval, provided as an embodiment of the present invention. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0038] This invention provides a document retrieval method that supports batch searching, see [link to related document]. Figure 1 This includes the following steps: (1) Using Elasticsearch as the search engine, the literature data is transformed into an ES index structure and relevant literature data is imported to generate an index and build an ES library (ES literature information library). In this implementation, the document index structure adopts a field mapping strategy, as shown in Table 1: Text fields such as Title and Abstract are set to text type with a keyword type subfield, and the ik_max_word segmenter is configured to simultaneously satisfy both precise and fuzzy word segmentation matching methods.

[0039] The Identifier and DOI text fields are set to keyword type because they are unique identifiers for documents and require exact value matching. The Identifier is an auto-incrementing document ID generated within the system, which can be used for sorting the document result set and setting page offsets.

[0040] Setting the PublicationYear and Pageinfo numeric fields to number type allows for matching and searching within a range of values.

[0041] Table 1 (2) Divide each index into multiple primary shards, and distribute the literature data to different primary shards through hash routing; each primary shard has at least one replica (replica shard). This embodiment optimizes ES cluster performance by reasonably configuring the number of shards and replicas and enabling query caching.

[0042] A shard is a physical unit for dividing data. Each index can be divided into multiple primary shards. Data is distributed across different shards via hash routing, achieving horizontal scaling, parallel read / write operations, and load balancing. The number of primary shards is determined by the data volume and the number of nodes, typically with a single shard size controlled between 10-50GB to avoid too many small shards or too few large shards. For example, if the literature data volume is 100 million records, the total data size is around 150GB. Using a standard shard size of 30GB, a total of 5 shards can be created.

[0043] Replicas are copies of the primary shard, used to improve data redundancy and query performance. When the node hosting the primary shard fails, a replica can automatically be promoted to primary, ensuring service continuity. Replicas can also distribute read requests, significantly improving query throughput by processing searches in parallel across multiple nodes. For example, setting the number of replicas to 1 ensures data redundancy while avoiding excessive space consumption by too many replicas. Disabling replicas during the initial data import and enabling them afterward greatly speeds up the data import process.

[0044] Preferably, enabling caching mechanisms such as query cache and request cache can accelerate repeated queries and significantly improve the response speed of batch retrieval.

[0045] (3) Based on the search target, construct a tree-like nested Boolean query. Parallel query is used during the query, and only the document ID is queried. The queried document IDs are stored in Redis Bitmap. (4) When a user views the results in pages, the corresponding page document ID is retrieved from the Redis Bitmap, and the corresponding document data is read from the ES database based on the document ID.

[0046] In practice (the processing of single search and batch search is the same, so it will not be elaborated here), batch search refers to merging multiple single search requests into a single search request. These single searches can correspond to different matching methods for different document fields, and in the result set of this merged single search request, the document records are not repeated and are arranged in order.

[0047] To simplify the illustration, the batch search in this article specifically refers to retrieving tens of thousands of documents with fuzzy matching conditions for a specific string field in the literature records at once. For example, retrieving all documents whose titles match conditions such as "...mathematics...", "...development...", "...interdisciplinary...", "...genetic engineering...", etc.

[0048] Therefore, the system supports users uploading a CSV file with only one column. The header row, or first row, represents the document field to be matched, in this case, "Title." Each subsequent row represents a fuzzy matching condition. As shown in Table 2: Table 2 Filtering invalid search terms: In batch search scenarios, short phrase filtering is a key step in improving search accuracy. The core reason for this strategy is that overly short search terms have vague semantic meanings, resulting in a large number of results with low relevance, failing to meet the need for precise document retrieval. For example, using only short phrases like "AI" or "A Project" as search terms may return hundreds of thousands of documents, but most of them are weakly related to the user's actual needs, wasting system computing resources and increasing the time cost for users to filter valid documents. For instance, when a single search term is a short phrase of less than 3 or 4 words, it will be automatically filtered out and not used as an actual search term. By filtering short search terms, search efficiency and result quality can be significantly improved, ensuring the practical value of batch searches.

[0049] Elasticsearch Boolean queries can efficiently combine a large number of search conditions, and can implement complex logic through must, should, filter, and must_not. It supports multi-level nesting and is a core tool for building high-precision batch search systems.

[0050] The total number of conditions in an Elasticsearch Boolean query is limited by the `indices.query.bool.max_clause_count` parameter of the Elasticsearch service. This limit is designed to prevent queries from becoming too large and consuming excessive CPU and memory resources. By default, this limit is 1024, meaning that the total number of clauses (such as conditions in `must`, `should`, `filter`, and `must_not`) in a single Boolean query cannot exceed this value.

[0051] When the total number of Boolean query conditions exceeds the limit, a too_many_clauses error will be encountered. This parameter is difficult to quantify to an accurate value. To make full use of the high-configuration hardware resources and optimized sharded replica design of this service, this application redesigns the condition combination logic and uses nested Boolean queries to circumvent the limit.

[0052] In the specific implementation, see Figure 2 Taking a search condition set of tens of thousands as an example, a two-level nested Boolean query structure can be set up, namely a root Boolean query and multiple sub-Boolean queries. In detail, the huge search condition set is grouped into groups of 100 conditions each, with each group corresponding to a sub-Boolean query. Each search condition within a group is a should statement, and then each sub-Boolean query uses the should statement as a condition of the root Boolean query.

[0053] If the set of search criteria is larger, more levels are needed. With this tree-like nested structure, each level handles only a limited number of subqueries. Given sufficient resources, the system can handle any number of search criteria while maintaining manageability and execution efficiency.

[0054] Since the result set of literature in the search results is very large, storing each search result in a result table or result file would be a huge waste of storage space and processing efficiency. Therefore, a Redis Bitmap is created to store all the document IDs of the search. Since one bit can represent one ID, for the example of 100 million documents, only about 10MB of space is needed to record all the document IDs of the search results.

[0055] Specifically, parallel queries are used during the query process, and only the document ID is queried. The retrieved document ID is then stored in a Redis Bitmap.

[0056] When a user views the results in pages, the corresponding page document ID is retrieved from the Redis Bitmap, and the corresponding document details are read from the ES database in real time.

[0057] Because only document IDs are stored, there is no need to save the massive document result sets for each search, significantly reducing storage consumption. Furthermore, retrieving document details based on document IDs is extremely fast, improving the user's experience and continuity when viewing results.

[0058] The query uses parallel query and only queries document IDs, with the goal of retrieving all document IDs that meet the search criteria from the Elasticsearch database.

[0059] Since the full result set contains a large number of data entries, there is a limit to the number of result sets that Elasticsearch can retrieve each time. Therefore, to obtain all the document IDs, Elasticsearch's search_after deep pagination method must be used to retrieve them step by step in multiple steps.

[0060] This application uses a unique, ordered document ID field as the sorting basis and combines it with Elasticsearch's `search_after` mechanism to achieve deep pagination retrieval. However, `search_after` retrieves a certain number of matching results starting from a certain offset. While it guarantees no duplicates and that the results are ordered, it has a core drawback: it's a unidirectional cursor mechanism. It must traverse each page sequentially, using the last `sort` value of the previous page as the starting offset for the next page. It cannot directly jump to page 500; it must start from page 1 and retrieve the `sort` values ​​of the first 499 pages sequentially. Furthermore, the default limit for the result set that the Elasticsearch engine can return per retrieval request is 10,000 records. This means that to obtain all matching results for a single search, it's necessary to perform multiple searches, each retrieving 10,000 results sequentially from beginning to end.

[0061] To address the low parallelism issue caused by serial queries in `search_after`, this application divides the Elasticsearch library into several data intervals. Each data interval uses a previously constructed tree-structured nested Boolean query as the search parameter, simultaneously querying document IDs from several data intervals. The details are as follows: This application pre-divides the Elasticsearch (ES) database into several data intervals. The division is based on the maximum number of parallel queries that the ES nodes in the system can support, multiplied by the expected total number of query batches, to obtain the total number of data intervals. Then, the entire data in the ES database is evenly divided according to the total number of data intervals.

[0062] For example, if an Elasticsearch (ES) database contains 100 million documents, and the maximum number of parallel paginated queries supported by ES node sharding and replica count is 50, then dividing the expected total retrieval time by the single pagination retrieval time yields a total of 2 query batches. This results in 100 data intervals. Therefore, each data interval can be divided into 100 intervals, with 1 million documents per interval. After division, each interval is assigned a fixed starting document ID. See [link to relevant documentation]. Figure 3 For example, the first interval starts with document 0, the second interval starts with document 1 million, the third interval starts with document 2 million, and so on.

[0063] Then, during pagination queries, pagination query requests are initiated in parallel for 50 data intervals, executed in two batches for a total of 100 pagination queries. The starting cursor for each `search_after` pagination query is the starting document ID of that data interval. This means that the results of each pagination query may overlap; for example, the 10,000 results for interval 1 may include documents from interval 2, or even documents from interval 3 and beyond.

[0064] However, thanks to the highly efficient deduplication processing of Redis Bitmap mentioned earlier, even if the results of parallel queries have duplicates and overlaps, they can be easily deduplicated when stored in Redis Bitmap. At the same time, since Redis Bitmap itself uses bit positions to replace document IDs, it can naturally perform sorting of the results documents.

[0065] To enhance the parallel execution capability of deduplication, this application employs a tree-like multi-level organization for the Redis Bitmap. Specifically, the results of every 10 data ranges are merged into the first-level Redis Bitmap, resulting in a total of 10 Redis Bitmaps in the first level. After all deduplication in the first level is completed, the results from the first level are merged into a single final result Redis Bitmap in the second level. This approach reduces contention for deduplication resources when aggregating parallel query results, thereby enhancing the parallelism of the deduplication operation.

[0066] The core of this invention is to break the dependency between pages in a paginated query, predetermine the starting offset of each page, and obtain the final result set through efficient deduplication and sorting methods, thereby realizing the parallel execution of paginated retrieval.

[0067] The present invention also provides a document retrieval device that supports batch retrieval, including one or more processors, for implementing the above-described document retrieval device that supports batch retrieval.

[0068] See Figure 4The present invention provides a document retrieval device that supports batch retrieval, comprising one or more processors, for implementing a document retrieval method that supports batch retrieval as described in the above embodiments.

[0069] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0070] An embodiment of the document retrieval device supporting batch retrieval of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any data processing device, including a document retrieval device supporting batch retrieval according to the present invention. Except for... Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0071] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0072] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0073] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described document retrieval method supporting batch retrieval.

[0074] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described document retrieval method supporting batch retrieval.

[0075] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A document retrieval method supporting batch retrieval, characterized in that, include: Using Elasticsearch as the search engine, the literature data is transformed into an ES index structure and relevant literature data is imported to generate an index and build an ES library; Each index is divided into multiple primary shards, and the literature data is distributed to different primary shards through hash routing; Each primary shard has at least one replica; Based on the search target, construct a tree-structured nested Boolean query, use parallel querying, and only query the document ID. Store the retrieved document IDs in a Redis Bitmap. When a user views the results in pages, the corresponding page document ID is retrieved from the Redis Bitmap, and the corresponding document data is read from the ES database based on the document ID. The query uses parallel querying and only queries the document ID, specifically: The Elasticsearch library is divided into several data ranges, and the document IDs are queried from each of the several data ranges according to the constructed tree-like nested Boolean query.

2. The method according to claim 1, characterized in that, Transforming literature data into an Elasticsearch index structure includes: Set the Title and Abstract text fields in the literature data to text type with keyword type subfields, and configure the ik_max_word word segmenter to meet both precise and fuzzy word segmentation matching methods. Set the Identifier and DOI text fields in the literature data to keyword type; The PublicationYear and Pageinfo numeric fields are set to number type to satisfy the matching retrieval of range values.

3. The method according to claim 1, characterized in that, It also includes enabling a caching mechanism to speed up repeated queries.

4. The method according to claim 1, characterized in that, Based on the search objective, several search criteria are generated and filtered. A tree-like nested Boolean query is then constructed based on the filtered search criteria.

5. The method according to claim 4, characterized in that, The search criteria are filtered out as follows: if the number of keywords for a single search criterion is less than a preset threshold, then that search criterion is filtered out.

6. The method according to claim 1, characterized in that, A tree-structured nested Boolean query consists of several levels of Boolean queries, with the last level consisting of several search conditions; the level is determined by the total number of conditions in an ES Boolean query and the number of search conditions.

7. The method according to claim 1, characterized in that, The Elasticsearch (ES) database is divided into several data intervals. Specifically, the maximum number of parallel queries that an ES node can support is multiplied by the expected total number of query batches to obtain the total number of data intervals. Then, the entire data in the ES database is divided equally according to the total number of data intervals. It also includes setting up a tree-like multi-level organization for Redis Bitmap, specifically: the results of every n data ranges are merged into the first-level Redis Bitmap, resulting in several Redis Bitmaps; the results of the n Redis Bitmaps in the first level are merged into the second-level Redis Bitmap, until the last level yields a final result Redis Bitmap.

8. A document retrieval device supporting batch retrieval, characterized in that, It includes one or more processors for implementing a document retrieval system supporting batch retrieval as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a document retrieval method supporting batch retrieval as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements a document retrieval method supporting batch retrieval as described in any one of claims 1-7.

Images