OFD archive file retrieval method and system

Abstract

The invention provides an OFD archive file retrieval method and system, and the method comprises the steps: receiving file data, and carrying out the authentication verification of the file data; splitting the file into file units and storing the file units into a message queue for asynchronous processing in sequence; extracting text information, table information and picture information in the OFD file to generate comprehensive text information; preprocessing the comprehensive text information, and recording the occurrence frequency of each segmented word and the corresponding page number; creating an index in a search engine cluster, wherein an index field comprises a file ID (Identity), word segmentation content, occurrence frequency and a page number; searching according to an input keyword, and arranging results in a descending order according to the occurrence frequency of word segmentation; and marking the page number of the keyword for retrieval and positioning. The method has the beneficial effects that rapid analysis and preprocessing of large-scale OFD data are realized through asynchronous processing of the message queue and an OCR (Optical Character Recognition) technology; redundancy is reduced through coarse-grained word segmentation during indexing, and recall is enhanced through fine-grained word segmentation during retrieval; the keyword granularity input by the user is adapted, and the retrieval scene adaptability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data retrieval, and in particular relates to a retrieval method and system for OFD archive files. Background Art

[0002] With the acceleration of digitization, electronic documents are increasingly used in archival management. As a national standard, the OFD (OpenFixed-layout Document) format demonstrates great potential in archival preservation due to its fixed format, long-term preservation capabilities, and security. However, with the rapid growth in the number of archives, efficient retrieval and management of these documents has become a critical issue.

[0003] Existing archival document retrieval systems often rely on simple keyword matching or search other file types, such as PDFs, but lack full-text search capabilities for OFD files. Furthermore, existing systems can encounter performance bottlenecks when processing large-scale archival data, resulting in slow retrieval speeds and low accuracy. Therefore, it is necessary to propose a new retrieval method and system for OFD archival documents to address these issues. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a retrieval method and system for OFD archive files, which are particularly suitable for fast and accurate full-text retrieval of OFD files.

[0005] The technical solution adopted by the present invention is as follows: First, a method for retrieving OFD archive files is provided, comprising the following steps:

[0006] Receive case file data and perform authentication and verification on the case file data;

[0007] Split the case file data into case file units and store them in the message queue for asynchronous processing in sequence;

[0008] Extract text information, table information and image information from OFD files and process them to generate comprehensive text information;

[0009] After pre-processing the comprehensive text information, the number of occurrences of each word and the corresponding page number are recorded;

[0010] Create an index in the search engine cluster. The index fields include document ID, word segmentation content, number of occurrences, and page number.

[0011] Search based on the keywords you entered, and sort the results in descending order based on the number of times the words appear;

[0012] Mark the page number where the keyword is located for search positioning.

[0013] Furthermore, processing the image information includes the following steps:

[0014] Convert the RGB value of the image information into a grayscale image according to the weight;

[0015] Set the threshold to generate a black and white image;

[0016] Remove noise through median filtering.

[0017] Furthermore, preprocessing of the comprehensive text information includes word segmentation and filtering.

[0018] Furthermore, an index configuration is created in the search engine cluster such that a coarse-grained word segmenter is used to split text during indexing, and a fine-grained word segmenter is used to parse query keywords during retrieval.

[0019] Furthermore, the search engine adopts Elasticsearch.

[0020] In a second aspect, a retrieval system for OFD archive files is provided, comprising:

[0021] Data receiving and processing module, used to receive external case file data and verify its legitimacy;

[0022] OFD parsing module, used to parse the contents of OFD files and extract searchable data from text, tables and images;

[0023] Index building module, used to build distributed full-text indexes for OFD files;

[0024] The search navigation module is used to provide retrieval interface and result optimization functions.

[0025] Furthermore, the data receiving and processing module includes:

[0026] Authentication unit, used to verify the legitimacy of the data source;

[0027] A data splitting unit is used to split batch data into individual case file units;

[0028] Message queue processing unit, used to process case file data through asynchronous queues to improve the system's concurrency capabilities;

[0029] The file download unit is used to download OFD files in multiple threads and supports retry on failure.

[0030] Furthermore, the OFD parsing module includes:

[0031] A table processing unit, used to extract text content in a table;

[0032] Image preprocessing unit, used to perform grayscale, binarization and noise reduction operations on the image;

[0033] OCR processing unit, used to recognize text content in images;

[0034] The word segmentation and statistics unit is used to segment and remove duplicates from text data, and record the number of occurrences and page numbers of the segmented words.

[0035] Furthermore, the index building module includes:

[0036] Index cluster unit, used to store index data through the Elasticsearch cluster;

[0037] The word segmenter configuration unit is used to use coarse-grained word segmentation during indexing and fine-grained word segmentation during retrieval;

[0038] The index field mapping unit is used to define the index structure of file ID, case ID, word segmentation content, number of occurrences and page number.

[0039] Furthermore, the search navigation module includes:

[0040] The query interface unit is used to receive precise query and fuzzy query requests input by the user;

[0041] Sorting unit, used to sort the results by weight according to the number of times the segmented words appear;

[0042] Navigation unit, used to mark the page number where the keyword appears, supports clicking to jump to the corresponding location in the OFD file.

[0043] The advantages and positive effects of the present invention are as follows: due to the adoption of the above technical solution, through asynchronous processing of message queues and OCR technology, large-scale OFD data can be quickly parsed and preprocessed; based on the descending sorting of the number of word occurrences, high-frequency matching documents are displayed first; the page numbers where the keywords are located are marked to reduce the user's search time in long documents; coarse-grained word segmentation during indexing reduces redundancy, and fine-grained word segmentation during retrieval enhances recall; the granularity of the keywords input by the user is adapted to improve the adaptability of the retrieval scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the method steps of an embodiment of the present invention

[0045] Figure 2 This is a schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present disclosure is described more fully below with reference to the accompanying drawings, which illustrate exemplary embodiments of the present disclosure. The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0047] like Figure 1 As shown, the present invention provides a method for retrieving OFD archive files, comprising the following steps:

[0048] S100, receiving case file data and performing authentication and verification on the case file data;

[0049] Authentication is achieved through API key verification (verifying the X-API-Key field in the request header) or digital signature (using the SHA-256 algorithm to generate a signature on the request body). At the same time, a hash check (MD5 or SHA-1 checksum) is performed on the case file data to ensure data integrity. If the verification fails, the request is rejected and a log is recorded.

[0050] S200, splitting the case file data into case file units and storing them in a message queue for asynchronous processing in sequence;

[0051] Based on the preset sharding strategy or time window sharding, batch data is split into independent file units and stored in a message queue (RabbitMQ or Kafka) for asynchronous processing. Data reliability is guaranteed through message persistence and ACK confirmation mechanism.

[0052] S300, extracting text information, table information and image information from the OFD file and processing them to generate comprehensive text information;

[0053] Parse the content structure of the OFD file, extract plain text from the specified tag position, use cell coordinate positioning to completely read the cross-page table content in the table area, optimize the image part (convert to grayscale, adjust black and white contrast, remove noise), and call the image text recognition engine to extract the text in the image. Finally, merge all text, tables and image content into unified text data.

[0054] S400, recording the number of occurrences of each word and the corresponding page number after pre-processing the comprehensive text information;

[0055] Use a word segmentation tool to split the merged text into words (allowing the loading of a custom dictionary to improve the recognition rate of professional terms), count the total number of occurrences of each word and the page number where it first appears, and at the same time filter out invalid words (such as meaningless words like "de", "le", etc.) and sensitive information (such as replacing the ID card number with a placeholder), and generate index data containing words, frequencies, and page numbers.

[0056] S500. Create an index in the search engine cluster. The index fields include file ID, segmented content, number of occurrences, and page number.

[0057] In a distributed search engine cluster, set the unique file identifier, the content of the words after segmentation, the number of occurrences of the words, and the page number where they are located as the core fields of the index, and sort them dynamically according to the weight of the words (for example, high-frequency words are prioritized). The index data is evenly distributed to multiple server nodes according to the case file identifier to improve the query efficiency.

[0058] S600. Retrieve according to the input keywords and sort the results in descending order of the number of occurrences of the segmented words.

[0059] The system matches the documents containing the word from the index and sorts them in descending order of the number of occurrences of the word in the document (if the number of occurrences is the same, sort them according to the document update time), and returns a list of retrieval results including the document title, brief content, and highlighted keywords.

[0060] S700. Mark the page number where the keyword is located for retrieval positioning.

[0061] Adopt the above method, and through asynchronous processing of the message queue and OCR technology, achieve fast parsing and preprocessing of large-scale OFD data; sort in descending order of the number of occurrences of the segmented words and preferentially display the documents with high-frequency matches; mark the page number where the keyword is located to reduce the search time of users in long documents.

[0062] To solve the problem of high OCR recognition error rate caused by problems such as many image noise points and uneven illumination in OFD files, an implementation method is provided in this embodiment.

[0063] In one embodiment, the processing of the picture information includes the following steps:

[0064] Convert the RGB values of the picture information into a grayscale image according to the weight.

[0065] In the process of converting a color image into a grayscale image, for each pixel point in the image, first obtain the color information of its red (R), green (G), and blue (B) channels, and set corresponding weight coefficients for these three channels respectively. Calculate the grayscale value of the pixel point through the sum of the products of these weighting coefficients and the corresponding color channel values.

[0066] Set a threshold to generate a black and white image.

[0067] A threshold is selected based on the specific conditions and requirements of the image. For each pixel in the grayscale image, its grayscale value is compared with the set threshold. If the grayscale value of a pixel is higher than the threshold, the pixel is set to white; otherwise, it is set to black.

[0068] Remove noise through median filtering.

[0069] Select a sliding window in the image. For each window, collect the grayscale values ​​of all pixels in it and sort these values; replace the original grayscale value of the center pixel of the window with the sorted middle value.

[0070] The above method is used to convert the image into high-contrast, low-noise input data through grayscale conversion, binarization and noise reduction; avoid missed detection or false detection, and completely extract the text content in the image.

[0071] In one embodiment, preprocessing of the comprehensive text information includes word segmentation and filtering.

[0072] In order to solve the problem of imbalance between retrieval efficiency and accuracy caused by too coarse or too fine granularity during index construction, an implementation method is provided in this embodiment.

[0073] In one embodiment, an index is created in a search engine cluster and configured to use a coarse-grained word segmenter to split text during indexing and a fine-grained word segmenter to parse query keywords during retrieval.

[0074] Using the above method, coarse-grained word segmentation reduces redundancy during indexing, and fine-grained word segmentation enhances recall during retrieval; it adapts to the granularity of keywords entered by users and improves adaptability to retrieval scenarios.

[0075] In one embodiment, the search engine adopts Elasticsearch.

[0076] like Figure 2 As shown, in order to facilitate the use of the OFD archive file retrieval method provided by the present disclosure, the present disclosure also provides an OFD archive file retrieval system, including:

[0077] The data receiving and processing module 10 is used to receive external case file data and verify its legitimacy;

[0078] OFD parsing module 20, used to parse the OFD file content and extract searchable data from text, tables and images;

[0079] An index building module 30 is used to build a distributed full-text index for OFD files;

[0080] The search navigation module 40 is used to provide a search interface and result optimization functions.

[0081] In one embodiment, the data receiving and processing module includes:

[0082] Authentication unit, used to verify the legitimacy of the data source;

[0083] A data splitting unit is used to split batch data into individual case file units;

[0084] Message queue processing unit, used to process case file data through asynchronous queues to improve the system's concurrency capabilities;

[0085] The file download unit is used to download OFD files in multiple threads and supports retry on failure.

[0086] In one embodiment, the data receiving and processing module includes:

[0087] Authentication unit, used to verify the legitimacy of the data source;

[0088] A data splitting unit is used to split batch data into individual case file units;

[0089] Message queue processing unit, used to process case file data through asynchronous queues to improve the system's concurrency capabilities;

[0090] The file download unit is used to download OFD files in multiple threads and supports retry on failure.

[0091] In one embodiment, the OFD parsing module includes:

[0092] A table processing unit, used to extract text content in a table;

[0093] Image preprocessing unit, used to perform grayscale, binarization and noise reduction operations on the image;

[0094] OCR processing unit, used to recognize text content in images;

[0095] The word segmentation and statistics unit is used to segment and remove duplicates from text data, and record the number of occurrences and page numbers of the segmented words.

[0096] In one embodiment, the index building module includes:

[0097] Index cluster unit, used to store index data through the Elasticsearch cluster;

[0098] The word segmenter configuration unit is used to use coarse-grained word segmentation during indexing and fine-grained word segmentation during retrieval;

[0099] The index field mapping unit is used to define the index structure of file ID, case ID, word segmentation content, number of occurrences and page number.

[0100] In one embodiment, the search navigation module includes:

[0101] The query interface unit is used to receive precise query and fuzzy query requests input by the user;

[0102] Sorting unit, used to sort the results by weight according to the number of times the segmented words appear;

[0103] Navigation unit, used to mark the page number where the keyword appears, supports clicking to jump to the corresponding location in the OFD file.

[0104] The following describes the contents involved in the above embodiment in conjunction with a preferred embodiment.

[0105] This system provides an interface for receiving case file data pushed from various authenticated systems. Received data is first split into individual files and sent to a message queue, currently using RabbitMQ for asynchronous processing and peak load shaving, increasing the system's concurrent processing capacity. The system then listens to the queue to sequentially read and process archived case file data. Basic case file information, along with associated business information and file information, is stored in the database's case file information table, business information table, and file record table through entity mappings based on the MyBatis framework. A receiving log table is also added. For data that fails to be stored, a re-storage function is provided to prevent process failures and ensure that the pushed data is fully stored. Archived OFD attachments are scanned through the file record table by a process. Undownloaded files are downloaded from the provider to the contract system's file server using a multi-threaded process, using the file address. A retry function is provided for failed attachment downloads to ensure complete archiving of the basic archive data.

[0106] Scan the file record table. For deleted files, delete the file data from the Elasticsearch index based on the file ID to reduce data redundancy. For newly added files, extract the file data. Using the open-source OFD toolkit ofd-lib, extract the title, page text, page number, and resource images from the OFD file. For images, extract the text content. First, perform image preprocessing, including grayscale conversion, binarization, and noise reduction. Grayscaling removes color information from the image, retaining only brightness information. Specifically, extract the image pixels and obtain the red (R), green (G), and blue (B) color values. A weighted average method (gray = R*n1% + G*n2% + B*n3%, where n1+n2+n3=1) is used to calculate the grayscale value of each pixel. A new image is created from these grayscale values. Image binarization involves mapping the pixel values ​​of a grayscale image to black and white. Specifically, a threshold is set; all pixel values ​​below the threshold are set to 0 (black), and pixel values ​​above or equal to the threshold are set to 255 (white), resulting in a distinct black and white image. Image denoising involves removing noise from an image to improve image quality. Specifically, the median of all pixel values ​​within the image is used to replace the central pixel value, generating a new image. Text data from the image is then extracted using optical character recognition (OCR), a technology that converts text in an image into editable text. The specific steps include text region detection, character segmentation, feature extraction, and character recognition to obtain the text data. Text region detection involves detecting text regions in an image and identifying continuous black or white characters. Character segmentation involves segmenting the detected text region into individual characters using the connected component method. Feature extraction involves using algorithms such as Sobel and Canny to detect edges and extract features from the segmented characters. Character recognition involves using a machine learning model to identify each character based on the extracted features, thereby extracting text data from the image. A natural language processor (NLP) is then used to perform word segmentation on the unified text data extracted from the OFD file. Text preprocessing involves loading the model and performing word segmentation to obtain a word segmentation array. Text preprocessing involves removing unnecessary spaces, line breaks, and special characters from the read text data. The model is currently loaded, and the word segmenter model provided by Apache OpenNLP is selected and loaded into the word segmenter to provide a basis for word segmentation. Word segmentation is currently performed using the API provided by Apache OpenNLP. The preprocessed text is passed to the word segmentation function, which then performs word segmentation based on the selected model logic, resulting in a word segmentation array, thus completing word segmentation extraction.Finally, for the extracted word segmentation, first filter the sensitive words, and then pass the HasMap<String,Integer> De-duplicate and count the words, and record the page number where each word appears. This will provide the basic file data for creating the Elasticsearch index later.

[0107] Build an Elasticsearch cluster and distribute each request to a different server. Then, install the IK Chinese word segmentation plugin, analysis-ik, and create a global query index to complete the basic environment. Create a full-text index field with type text, insert the parser analyzer set to ik smart (coarsest granularity), and query parser search_analyzer set to ik_max_word (finest granularity), for fields such as file ID, file name, content segmentation, number of segmentation occurrences, page number on which the segmentation occurs, and the associated case ID and case name. Store the field values ​​to complete the index creation.

[0108] Based on existing web technologies, a searchable list query page is created. Then, through the interface provided by the Elasticsearch toolkit, precise, fuzzy, and multi-word combination queries are performed on the Elasticsearch index fields, and the results are returned to the list query page. After completing the word segmentation and deduplication for each file and adding a record of the number of occurrences, when querying file content, the returned results are sorted from high to low by the number of occurrences. For each item in the returned list, a preview function for the file details and a navigation to the keywords that appeared in the file are provided. Users can use keywords to navigate to the corresponding page of the article where the keyword appears.

[0109] Based on the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0110] An electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the OFD archive file retrieval method provided by the present disclosure.

[0111] Electronic device is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device may also refer to various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0112] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the OFD archive file retrieval method provided by the present disclosure.

[0113] Various embodiments of the present disclosure may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the OFD archive file retrieval method provided in the present disclosure.

[0115] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for retrieving OFD archive files, characterized in that: The following steps are involved: Receive case file data and perform authentication and verification on the case file data; Split the case file data into case file units and store them in the message queue for asynchronous processing in sequence; Extract text information, table information and image information from OFD files and process them to generate comprehensive text information; After pre-processing the comprehensive text information, the number of occurrences of each word and the corresponding page number are recorded; Create an index in the search engine cluster. The index fields include document ID, word segmentation content, number of occurrences, and page number. Search based on the keywords you entered, and sort the results in descending order based on the number of times the words appear; Mark the page number where the keyword is located for search positioning.

2. The OFD archive file retrieval method according to claim 1, characterized in that: Processing image information includes the following steps: Convert the RGB value of the image information into a grayscale image according to the weight; Set the threshold to generate a black and white image; Remove noise through median filtering.

3. The OFD archive file retrieval method according to claim 1, characterized in that: Preprocessing of comprehensive text information includes word segmentation and filtering.

4. The OFD archive file retrieval method according to claim 1, characterized in that: The index configuration created in the search engine cluster is to use a coarse-grained word segmenter to split the text during indexing and a fine-grained word segmenter to parse the query keywords during retrieval.

5. The OFD archive file retrieval method according to any one of claims 1 to 4, characterized in that: The search engine uses Elasticsearch.

6. A retrieval system for OFD archive files, characterized in that: include: Data receiving and processing module, used to receive external case file data and verify its legitimacy; OFD parsing module, used to parse the contents of OFD files and extract searchable data from text, tables and images; Index building module, used to build distributed full-text indexes for OFD files; The search navigation module is used to provide retrieval interface and result optimization functions.

7. The OFD archive file retrieval system according to claim 6, characterized in that: The data receiving and processing module includes: Authentication unit, used to verify the legitimacy of the data source; A data splitting unit is used to split batch data into individual case file units; Message queue processing unit, used to process case file data through asynchronous queues to improve the system's concurrency capabilities; The file download unit is used to download OFD files in multiple threads and supports retry on failure.

8. The OFD archive file retrieval system according to claim 6, characterized in that: OFD parsing modules include: A table processing unit, used to extract text content in a table; Image preprocessing unit, used to perform grayscale, binarization and noise reduction operations on the image; OCR processing unit, used to recognize text content in images; The word segmentation and statistics unit is used to segment and remove duplicates from text data, and record the number of occurrences and page numbers of the segmented words.

9. The OFD archive file retrieval system according to claim 6, characterized in that: Index building blocks include: Index cluster unit, used to store index data through the Elasticsearch cluster; The word segmenter configuration unit is used to use coarse-grained word segmentation during indexing and fine-grained word segmentation during retrieval; The index field mapping unit is used to define the index structure of file ID, case ID, word segmentation content, number of occurrences and page number.

10. The OFD archive file retrieval system according to claim 6, characterized in that: The search navigation module includes: The query interface unit is used to receive precise query and fuzzy query requests input by the user; Sorting unit, used to sort the results by weight according to the number of times the segmented words appear; Navigation unit, used to mark the page number where the keyword appears, supports clicking to jump to the corresponding location in the OFD file.

Images