A method and system for fast retrieval of archival material

Abstract

This invention relates to the field of data retrieval technology, specifically disclosing a method and system for rapid retrieval of archival materials. The method includes: using OCR technology to perform batch automated processing of images and structurally encapsulating them as raw data units; constructing an Elasticsearch (ES) cluster composed of multiple nodes; dividing the archival data index into multiple shards and distributing them across different nodes; performing mapping design during ES index creation, defining field types and analyzers; batch importing the raw data units into the ES index; performing full-text search and vector search according to the retrieval request, and fusing the results of the full-text search and vector search; controlling the number of ES retrieval results loaded each time; configuring a thread pool, setting dynamically adjustable parameters, and encapsulating concurrent retrieval requests as asynchronous tasks for unified scheduling. The technical solution of this invention can support the processing of massive amounts of power industry archival materials and improve retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of data retrieval technology, and in particular to a method and system for rapid retrieval of archival materials. Background Technology

[0002] In the power sector, historical records have largely existed in unstructured forms such as scanned paper documents and image files, and their quantity is enormous and growing rapidly. Traditional document retrieval methods rely on manual classification or simple filename matching, which presents four major application bottlenecks: First, unstructured documents (such as scanned PDFs and photos of handwritten documents) cannot be directly retrieved, requiring manual review of each document, which is extremely inefficient; second, OCR recognition only outputs character sequences, lacking semantic meaning related to power business, resulting in low retrieval accuracy; third, defects in the quality of the original image (noise, tilt) limit the upper limit of OCR recognition, making direct processing of the original image inefficient; and fourth, single-node storage systems struggle to support rapid retrieval of massive amounts of documents, single-threaded query processing cannot fully utilize system resources, application-layer concurrency capabilities are weak, and there is a lack of fine-grained access control, posing a risk of information leakage.

[0003] In existing technologies, simple OCR recognition technology can only convert images to text, lacking efficient search engine support; traditional relational databases, while capable of storing structured data, have poor full-text search performance and cannot handle fuzzy queries and complex condition combinations; some search systems use stand-alone search engines, lacking scalability and struggling to cope with scenarios of surging data volumes; some archival systems, while introducing a hybrid mode of full-text and phasor search, lack a large-scale model re-ranking process, relying solely on surface-level matching and ranking results, which fails to meet the high-precision retrieval needs of the power industry. Furthermore, archival retrieval must strictly adhere to the State Grid's regulations and requirements for information and data security, ensuring traceability and controllable access for archival retrieval operations. With the ongoing digital transformation, enterprises' demands for "fast, accurate, comprehensive, and secure" archival retrieval are becoming increasingly urgent.

[0004] Therefore, there is an urgent need for a fast retrieval technology that can support the processing of massive amounts of power data and improve retrieval efficiency. Summary of the Invention

[0005] One of the objectives of this invention is to provide a method for rapid retrieval of archival data, which can support the processing of massive amounts of power archive data and improve retrieval efficiency.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution: A method for rapid retrieval of archival materials includes the following: The identification and correction steps involve establishing a standardized image preprocessing pipeline to preprocess all documents to be scanned before OCR recognition; using OCR technology to automatically process the preprocessed images in batches to obtain initial OCR recognition results; and associating the corrected text content with the document's metadata and encapsulating it in a structured manner as the original data unit. The steps to build a search engine include constructing an Elasticsearch cluster consisting of multiple nodes; dividing the archive data index into multiple shards and distributing them across different nodes; The index building steps include mapping design when creating the ES index, defining field types and analyzers, and importing raw data units into the ES index in batches. The full-text search process involves performing full-text and vector searches based on the search request, and then merging the results of the full-text and vector searches. The search result loading process controls the number of ES search results loaded each time. The concurrent retrieval process involves configuring a thread pool, setting dynamically adjustable parameters, and encapsulating concurrent retrieval requests into asynchronous tasks for unified scheduling.

[0007] Furthermore, it also includes recording steps, collecting and storing the recorded retrieval logs in real time.

[0008] Furthermore, in the identification and error correction steps, the pipeline preprocessing includes removing image noise, enhancing text outlines, correcting image tilt angles, and rotating the image according to the text direction.

[0009] Furthermore, in the search engine setup step, the ES cluster includes nodes for storing index data and nodes for coordinating query requests; The archive data index is divided into multiple shards and stored on different storage index nodes, and replica shards are set up.

[0010] Furthermore, in the identification and error correction steps, the document text is also subjected to topic analysis to generate topic tags as additional keyword fields; During the index building step, the keyword field is imported into the Elasticsearch index.

[0011] Furthermore, in the full-text retrieval step, the document rankings returned by full-text retrieval and vector retrieval are obtained respectively, and the comprehensive score of each document is recalculated according to the RRF algorithm to generate an optimized ranking; The titles, summaries, and full-text snippets of the top N documents returned after the RRF algorithm is fused, along with the user's original query, are sent to a pre-set large-scale language model. The large-scale language model then re-ranks the top N documents based on their semantic relevance and importance to the user's query, and this ranking is used as the final search result.

[0012] Furthermore, in the search result loading step, for the first 1000 search results, the `from` and `size` parameters of Elasticsearch are used to implement front-end pagination. For search results exceeding 1000, the SearchAfter parameter is used to implement pagination on the front end.

[0013] Furthermore, in the concurrent retrieval step, each query request is encapsulated as an asynchronous task and submitted to the thread pool; multiple asynchronous tasks are sent concurrently to the ES cluster, where multiple storage index data nodes process them in parallel and coordinate the aggregation of query request nodes.

[0014] The second objective of this invention is to provide a rapid retrieval system for archival materials using the above-described method.

[0015] This solution constructs an Elasticsearch (ES) cluster consisting of multiple nodes and distributes the archival data index into multiple shards across different nodes, overcoming the performance bottleneck of single-node storage systems. It supports the storage and processing of massive amounts of power archive data and achieves load balancing and horizontal scaling through a sharding mechanism, ensuring system stability and scalability when facing surges in data volume. Furthermore, the solution employs a hybrid retrieval mode combining full-text search and vector search during the retrieval phase, fusing the results of both to significantly improve the recall and precision of archival retrieval, avoiding missed or false positives that might occur with a single retrieval method. Finally, by configuring a thread pool, setting dynamically adjustable parameters, and encapsulating concurrent retrieval requests as asynchronous tasks for unified scheduling, while controlling the number of ES retrieval results loaded each time, the solution effectively addresses the slow response problem caused by system resource contention in high-concurrency scenarios. This fully utilizes system resources, significantly improving the system's concurrent processing capabilities and retrieval response speed, thereby improving the overall efficiency of archival data retrieval and the user experience. Attached Figure Description

[0016] Figure 1 This is a flowchart of an embodiment of a method for rapid retrieval of archival materials. Detailed Implementation

[0017] The following detailed description illustrates the specific implementation method: Example 1 like Figure 1 As shown in this embodiment, a method for rapid retrieval of archival data includes the following: S1. Recognition and Error Correction Steps. Establish a standardized image preprocessing pipeline to preprocess all documents to be scanned, including images (e.g., JPEG, PNG formats) and PDF scans, before OCR recognition.

[0018] In this embodiment, the pipeline integrates a series of image processing algorithms based on the OpenCV library. Specifically, it removes image noise through algorithms such as Gaussian filtering and bilateral filtering, enhances text outlines through sharpening, automatically corrects the image tilt angle based on layout analysis, and rotates the image according to the text direction, providing clear and regular image input for downstream OCR tasks and significantly improving the OCR recognition rate.

[0019] The pre-processed images are processed in batches using OCR (Optical Character Recognition) technology to obtain initial OCR recognition results. This embodiment also employs a large language model trained on a power knowledge base to place the initial OCR recognition results into a real power business context (such as power distribution and operation and maintenance scenarios) for semantic understanding and error correction, resulting in corrected text content. For example, the large language model automatically corrects misidentified "isolation start" to "isolation switch," ensuring the accuracy of professional terms such as "load switch."

[0020] The corrected text content is associated with the archive's metadata (such as the user ID, work order number, attachment identifier, and other power system customer information), and together they are structured and encapsulated into JSON data, which serves as the raw data unit for subsequent import into Elasticsearch.

[0021] S2. Search Engine Setup Steps. Build an Elasticsearch (ES) cluster consisting of multiple nodes to process massive amounts of electronic archival data. The ES cluster includes Data nodes (storing index data) and Coordinating nodes (coordinating query requests) to achieve high availability and load balancing.

[0022] The archive data index is divided into multiple shards and distributed across different Data nodes. Replica shards are configured to ensure that data is not lost and service is not interrupted in the event of a single node failure. For performance assurance, the Data nodes in this embodiment use SSDs to improve I / O performance and optimize JVM memory configuration (typically allocating 50% of the machine's memory) to cache frequently accessed index data, ensuring millisecond-level search response times.

[0023] S3. Index Building Steps. When creating the Elasticsearch index, perform mapping design, defining field types and analyzers. Specifically, for fields such as "User ID," "User Name," and "Power Supply Unit," set to "type":"keyword" for exact matching and aggregation; for fields such as "Full Text Content" recognized by OCR, set to "type":"text," and use the IK Chinese word segmenter (ik_max_word) for fine-grained word segmentation, while supporting exact matching through "fields":{"keyword":{"type":"keyword"}}.

[0024] The encapsulated raw data units are imported into the ES index in batches using the ES Bulk API. In step S1, this embodiment also performs topic analysis on the archive text using clustering algorithms such as K-Means, generating topic tags as additional keyword fields and storing them in the ES index as an additional retrieval dimension to optimize the retrieval mechanism.

[0025] S4. Full-text search steps. Based on the search request, perform full-text search and vector search. Use the RRF algorithm to fuse the results of the full-text search and vector search, improving the overall effect of keyword matching and semantic understanding.

[0026] Specifically, the document rankings returned by the two retrieval methods are obtained separately. Then, the overall score of each document is recalculated using the RRF algorithm to generate a final optimized ranking that balances keyword matching and semantic relevance. For example, for each document, the sum of the reciprocals of its ranking across all search results is calculated. The higher the ranking (e.g., number 1), the larger its reciprocal (1 / rank), and the higher its contribution to the score. Full-text search (based on keywords) returns one ranked list; vector search (based on semantics) returns another ranked list. The RRF algorithm merges these two ranking lists. A document that ranks highly in both lists will have a high overall RRF score; a document that ranks highly in one list but low in the other will have its score suppressed.

[0027] Full-text search involves creating text-type fields for file content, usernames, and other text fields, and applying appropriate word segmentation tools (such as the IK Chinese word segmenter). Elasticsearch uses `match` or `multi_match` queries for retrieval. Elasticsearch segments the query terms and searches for matching documents in the inverted index, then sorts them using its built-in BM25 relevance scoring algorithm.

[0028] Vector semantic retrieval refers to using an embedding model (such as m3e, bge, etc.) to convert the text content of each document into a high-dimensional vector and store it in the dense_vector type field of Elasticsearch. When a user enters a query, the same model is used to convert the query into a vector, and then Elasticsearch's k-nn query is used to calculate the cosine similarity between the query vector and the document vector, returning the most similar document.

[0029] In this embodiment, a large-scale model re-ranking is also introduced. Specifically, the titles, summaries, and full-text fragments of the Top N (e.g., Top 20) documents returned after the RRF algorithm fusion, along with the user's original query, are sent to a large-scale language model with an input instruction. The instruction, for example, requires the large-scale language model to act as an archival expert and re-rank the Top N documents based on their semantic relevance and importance to the user's query.

[0030] Retrieve the reordered list of document IDs and brief explanations returned from the large language model. Use this list as the final search result ranking. This step greatly improves the relevance of the final results because it leverages the deep language understanding and reasoning capabilities of the large language model, which is difficult to achieve with simple keyword matching (BM25) and vector similarity.

[0031] In this embodiment, the following is also introduced: Boolean compound queries use Elasticsearch's boolean query to combine clauses such as must (must match), should (should match, improves relevance), and filter (filter, does not score) to handle users' complex multi-condition search needs.

[0032] Full-text fuzzy search allows you to use match or multi_match to query text fields such as "full text content". ES automatically segments the query terms and searches for matching documents in the inverted index. The built-in BM25 algorithm sorts the results by relevance, effectively expanding the search scope.

[0033] Access control is achieved by forcibly adding user permission conditions (e.g., {"term":{"orgNo":"user's power supply unit"}}) to the filter clause of a boolean query. Filter queries are automatically cached by Elasticsearch, resulting in extremely fast subsequent queries with the same permissions, ensuring data security without compromising performance.

[0034] S5. Search Result Loading Steps. Control the number of ES search results loaded each time to avoid slow system response or lag caused by loading too much data at once, thus improving user experience. Specifically, Shallow pagination (From / Size): For the first 1000 or fewer search results, use the `from` and `size` parameters in Elasticsearch to implement pagination on the front end. For example, `"from":0,"size":10` retrieves the first page.

[0035] Deep pagination (SearchAfter): For deep pagination requirements exceeding 1000 search results, use the SearchAfter parameter. The SearchAfter parameter mechanism uses the sort value of the last document in the previous page's results as a cursor to retrieve the next page. This performance is significantly higher than from / size, making it ideal for deep browsing of large datasets and avoiding performance bottlenecks.

[0036] S6. Concurrent Retrieval Steps. Configure a thread pool (ThreadPoolExecutor) at the application layer, set dynamically adjustable parameters, and encapsulate concurrent retrieval requests as asynchronous tasks for unified scheduling to reduce resource consumption; in this embodiment, each query request is encapsulated as an asynchronous task and submitted to the thread pool.

[0037] This embodiment utilizes Elasticsearch's asynchronous API to send multiple asynchronous tasks concurrently to the Elasticsearch cluster, making full use of its distributed parallel computing capabilities. Multiple Data nodes process the data in parallel, while Coordinating nodes aggregate the results. The collaboration between the two significantly reduces the time required for large-scale file retrieval and improves system response efficiency.

[0038] S7. Log recording steps. Use the Filebeat lightweight collector to collect and record retrieval logs in real time. The retrieval logs include fields such as keywords, user ID, retrieval time, hit count, and power supply unit permissions.

[0039] Logs are transmitted to Logstash via the lightweight Filebeat collector for parsing and formatting, and then stored in a dedicated Elasticsearch (ES) log index. The ELK Stack is a complete ecosystem suite centered around Elasticsearch, with Logstash being a core component that serves as a pipeline for data collection, parsing, transformation, and transmission.

[0040] Kibana is used to visualize and analyze data in log indexes, such as creating dashboards to monitor key metrics like popular search terms, user search habits, peak concurrent requests, and search success rates. This provides robust data support for iterative optimization of archive management strategies and search functions. Simultaneously, all logs provide tamper-proof and detailed records for security auditing. Kibana serves as a data visualization and management platform within the Elastic Stack, enabling the presentation of data stored in Elasticsearch through a graphical interface.

[0041] This embodiment also provides a rapid archival document retrieval system using the aforementioned method, including an image recognition module and a search engine. The image recognition module receives input archival document images, performs preprocessing such as noise reduction and text enhancement to obtain clear and well-organized image input, and performs semantic error correction using a large-scale error correction module based on a power knowledge base and a large model. The recognized text is placed in a real business context (such as power distribution and maintenance scenarios) for understanding and rewriting, ensuring the accuracy of key power-related terminology. The associated archival metadata is encapsulated in a JSON structured document. The search engine is deployed and built on an Elasticsearch cluster, using high-performance hardware to cache index data. It implements full-text intelligent search applications based on composite queries, hybrid retrieval, and access control. Through ES asynchronous API, pagination, and multi-threaded retrieval, it improves the efficiency of big data retrieval, shortens the processing time of parallel tasks, and solves the problem of weak multi-concurrency processing capabilities at the application layer.

[0042] This solution establishes a standardized image preprocessing pipeline before OCR recognition, effectively removing image noise, enhancing text outlines, and automatically correcting tilt angles. This provides a regular image input for subsequent OCR recognition, improving the accuracy of character recognition. Furthermore, based on a large language model, the OCR recognition results are placed within a real power business context for understanding and rewriting, with automatic corrections to ensure the accuracy of power archive data.

[0043] In terms of search ranking optimization, this solution adopts the RRF algorithm to integrate full-text and vector search results, and introduces a large language model to re-rank the Top N documents. This effectively makes up for the shortcomings of the traditional BM25 algorithm and vector similarity based only on surface feature matching, so that the search results presented to users are more accurate and relevant in terms of business logic.

[0044] Regarding system performance and data loading strategies, this solution employs differentiated pagination strategies for search results of different orders of magnitude. For large-scale data searches exceeding 1000 results, the SearchAfter parameter utilizes the sort value of the previous page as a cursor, avoiding the significant performance overhead and memory overflow risk associated with traditional deep pagination, thus ensuring system smoothness when deeply browsing massive archives.

[0045] This solution also integrates a comprehensive log recording and analysis mechanism, which not only provides data support for subsequent analysis of popular search terms and optimization of file management strategies, but also provides tamper-proof and detailed records for system security auditing, ensuring the traceability of file retrieval operations and meeting the stringent requirements of the power industry for information security and data compliance.

[0046] The above are merely embodiments of the present invention. The invention is not limited to the fields covered by these embodiments. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are able to access all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for rapid retrieval of archival materials, characterized in that, Includes the following: The identification and correction steps involve establishing a standardized image preprocessing pipeline to preprocess all documents to be scanned before OCR recognition; using OCR technology to automatically process the preprocessed images in batches to obtain initial OCR recognition results; and associating the corrected text content with the document's metadata and encapsulating it in a structured manner as the original data unit. The steps to set up a search engine include building an Elasticsearch (ES) cluster consisting of multiple nodes. The archive data index is divided into multiple fragments and stored separately on different nodes; The index building steps include mapping design when creating the ES index, defining field types and analyzers, and importing raw data units into the ES index in batches. The full-text search process involves performing full-text and vector searches based on the search request, and then merging the results of the full-text and vector searches. The search result loading process controls the number of ES search results loaded each time. The concurrent retrieval process involves configuring a thread pool, setting dynamically adjustable parameters, and encapsulating concurrent retrieval requests into asynchronous tasks for unified scheduling.

2. The method for rapid retrieval of archival materials according to claim 1, characterized in that: It also includes recording steps, collecting and storing retrieval logs in real time.

3. The method for rapid retrieval of archival materials according to claim 2, characterized in that: In the identification and error correction steps, the pipeline preprocessing includes removing image noise, enhancing text outlines, correcting image tilt angles, and rotating the image according to the text direction.

4. The method for rapid retrieval of archival materials according to claim 3, characterized in that: In the search engine setup steps, the ES cluster includes nodes for storing index data and nodes for coordinating query requests; The archive data index is divided into multiple shards and stored on different storage index nodes, and replica shards are set up.

5. The method for rapid retrieval of archival materials according to claim 4, characterized in that: In the identification and error correction steps, the document text is also subject to topic analysis to generate topic tags as additional keyword fields. During the index building step, the keyword field is imported into the Elasticsearch index.

6. The method for rapid retrieval of archival materials according to claim 5, characterized in that: In the full-text search step, the document rankings returned by full-text search and vector search are obtained respectively, and the comprehensive score of each document is recalculated according to the RRF algorithm to generate an optimized ranking. The titles, summaries, and full-text snippets of the top N documents returned after the RRF algorithm is fused, along with the user's original query, are sent to a pre-set large-scale language model. The large-scale language model then re-ranks the top N documents based on their semantic relevance and importance to the user's query, and this ranking is used as the final search result.

7. The method for rapid retrieval of archival materials according to claim 6, characterized in that: In the search result loading step, for the first 1000 or so search results, the `from` and `size` parameters of Elasticsearch are used to implement front-end pagination. For search results exceeding 1000, the SearchAfter parameter is used to implement pagination on the front end.

8. The method for rapid retrieval of archival materials according to claim 7, characterized in that: In the concurrent retrieval step, each query request is encapsulated as an asynchronous task and submitted to the thread pool; multiple asynchronous tasks are sent concurrently to the ES cluster, where multiple storage index data nodes process them in parallel and coordinate the aggregation of query request nodes.

9. A rapid retrieval system for archival materials, characterized in that: Use the method described in any one of claims 1-8.

Images