Multi-modal data intelligent extraction system based on OCR (Optical Character Recognition) and semantic analysis

Abstract

The technical scheme of the invention discloses a multi-modal data intelligent extraction system based on OCR (Optical Character Recognition) and semantic analysis, which adopts a'four-layer architecture design ', namely a data layer, a processing layer, a service layer and an application layer. According to the system disclosed by the invention, the stability and the input quality of OCR (Optical Character Recognition) recognition are improved through multi-modal data preprocessing, a semantic understanding mechanism is introduced in a recognition process to enhance the field positioning and extraction capability, and flexible adaptation to various document types is realized through a customizable field extraction template; therefore, the automation degree, accuracy and expansibility of document processing are improved, and the method is widely applicable to scenes such as document structured analysis, business information extraction, automatic data entry and intelligent auditing, and is particularly applicable to automatic identification and extraction of key fields of documents of multiple types and formats such as bills, contracts, forms, official documents, reports and the like.

Description

Technical Field

[0001] This invention relates to a system for extracting key fields from documents through multimodal OCR recognition and semantic analysis, belonging to the field of information processing and intelligent recognition technology, and particularly to the technical field of cross-application of computer vision, natural language processing and intelligent document processing. Background Technology

[0002] With the accelerating pace of informatization, enterprises, government departments, and military units need to process a large number of structured, semi-structured, and unstructured business documents every day. These documents come in various forms, including but not limited to invoices, contracts, official documents, forms, reports, receipts, etc. To automate business processing, key information in these documents (such as name, address, amount, time, contract number, client name, and contractor name) usually needs to be accurately extracted and stored in a structured format.

[0003] Current mainstream automated document processing technologies primarily rely on Optical Character Recognition (OCR) algorithms to recognize document content. Traditional OCR technology can detect and recognize text regions in image-based documents and output text results. However, these technologies typically focus only on extracting the text itself, and their ability to locate key information, perform structured parsing, and understand semantics in complexly formatted documents is relatively limited.

[0004] In recent years, some document parsing methods based on layout analysis (such as the LayoutLM series of models) have begun to combine visual and textual information to improve the accuracy of field extraction. These methods have achieved some success in document scenarios with clear rules and standardized formatting, but they still face the following main problems in practical business applications: (i) Insufficient multimodal information fusion: Most existing technologies rely on only a single modality (such as visual OCR results or pure text semantic information), lacking effective fusion of multidimensional information such as layout, font style, position features, and contextual semantics, resulting in a significant drop in recognition accuracy in documents with complex layout and variable fields.

[0005] (ii) Zero semantic understanding ability: Traditional OCR only stays at the level of text detection and recognition. It has a weak understanding of the "meaning of fields" and cannot distinguish the semantics of similar fields (such as "Party A" and "Purchase Party", "Party B" and "Supplier Party"), resulting in extraction confusion.

[0006] (iii) Poor adaptability and robustness: Many systems rely on predefined templates or specific layouts. When faced with new formats, layouts, or low-quality scanned documents, rules or templates need to be manually redefined, making it difficult to achieve large-scale, multi-scenario automated deployment.

[0007] For example, the published paper "Document Understanding with LayoutLM" (published in the Association for Computational Linguistics) proposes combining OCR results with layout features and using a pre-trained language model for document understanding. However, this type of method still has significant room for improvement in extraction accuracy and generalization ability when facing cross-domain, irregular layout, and semantically ambiguous scenarios.

[0008] In summary, existing technologies suffer from low accuracy in key field recognition, poor adaptability, insufficient semantic understanding capabilities, and strong template dependence when facing complex business scenarios. These technical shortcomings mean that a significant amount of manual proofreading is still required in practical applications, making it difficult to meet the demands of large-scale business automation and low-cost, high-efficiency document processing. Therefore, there is a need for an automatic key field extraction method that can integrate multimodal information, possess semantic understanding capabilities, and adapt to complex documents across multiple scenarios, in order to improve overall recognition accuracy and robustness while reducing the cost of manual intervention. Summary of the Invention

[0009] The technical problem this invention aims to solve is that existing document recognition and information extraction technologies mainly rely on OCR engines to perform text recognition on images or PDF files, but in complex scenarios, they still suffer from the following significant technical bottlenecks: First, in terms of multimodal data preprocessing, existing methods are not adaptable to different document formats: For documents from multiple sources such as PDF files, scanned documents, and photographed images, there are often interference factors such as seal occlusion, image blurring, uneven lighting, and background noise, which directly affect the recognition accuracy of OCR. Existing preprocessing methods are mostly limited to basic image binarization and rotation correction, lacking targeted intelligent preprocessing strategies such as seal detection, image enhancement, and noise suppression, making it difficult to effectively improve the quality of input images and the stability of subsequent recognition. Secondly, in terms of semantic understanding, traditional OCR engines only have character-level recognition capabilities and cannot understand the logical relationships and business semantics between texts. When dealing with documents with complex structures, inconsistent formats, and unfixed field positions, it is difficult to achieve accurate extraction of key fields by simply relying on OCR output. In addition, in terms of the ability to customize field extraction templates, existing systems mostly use fixed extraction logic and lack flexible and configurable field mapping and template management mechanisms. When faced with different types of documents (such as invoices, contracts, payment vouchers, etc.), it is often necessary to manually modify the rules or retrain the model to complete the field adaptation, resulting in high development and maintenance costs, slow response to new business scenarios, and difficulty in meeting diverse application needs.

[0010] To address the aforementioned technical problems, the present invention discloses a multimodal data intelligent extraction system based on OCR recognition and semantic analysis. Its key feature is the adoption of a four-layer architecture comprising a data layer, a processing layer, a service layer, and an application layer. This architecture achieves a closed-loop process from initial access to original business documents, intelligent parsing, semantic extraction, review processing, and structured output, enabling efficient and automated extraction and processing of multimodal business documents. The data layer is responsible for accessing and storing original business documents; The processing layer is responsible for basic parsing, including image preprocessing, layout analysis, OCR recognition, and model invocation. The service layer is responsible for providing core services including intelligent file parsing, customized annotation templates, batch intelligent recognition, data auditing and optimization, and structured output. The application layer is responsible for providing users with a multimodal data intelligent extraction system, enabling integration with external systems and business processes.

[0011] Preferably, the original business documents include contracts, payment vouchers, invoices, and bank acceptance bills.

[0012] Preferably, the multimodal data intelligent extraction system is implemented using the following steps: Step S1: Construct field extraction tasks for different document types and business scenarios to drive subsequent batch recognition and parsing processing. The system generates a unique task identifier and task parameter set for each extraction task. Subsequent recognition scheduling and semantic matching are driven by task configuration to achieve unified scheduling and parallel processing. Step S2: After receiving the multimodal business documents through the data acquisition module, the input files are standardized and preprocessed to output a unified intermediate file format for subsequent layout recognition and OCR calls. Step S3: Using a lightweight layout structure recognition algorithm, the preprocessed document is segmented into regions. The OCR engine is called to recognize the text in the located regions and output the text content and coordinate information. The recognition results are then semantically preprocessed using Prompt prompts. Step S4: Batch file intelligent recognition. The recognition task is processed in parallel through multi-process scheduling, and the system combines OCR recognition and semantic matching algorithms to automatically extract key fields. Step S5: Manual verification and log auditing; Step S6: The approved data is exported according to the target business system format. Users can customize field mapping rules, and the system supports direct database connection, writing the extracted results to the external database in real time to achieve business linkage.

[0013] Preferably, in step S1, two types of extraction task construction methods are supported: custom task construction and template task construction. To complete the custom task construction, users can directly define the field names, field types, and Prompt semantic prompts to be extracted in the task configuration interface. The system will then perform semantic matching and value extraction in the subsequent recognition process based on the defined field information, without the need to create a template in advance. In template task construction, when the document type is stable and the scenario is standardized, users can pre-create templates using annotation tools. The templates include the coordinate positions of fields, semantic keywords, and field keys. During the template task construction process, the system automatically loads the corresponding templates and binds the template content to the recognition task.

[0014] Preferably, in step S1, the task parameters include field definitions, task type, and document type identifier.

[0015] Preferably, in step S2, the preprocessing includes grayscale conversion, binarization, stamp detection, and background noise removal.

[0016] Preferably, in step S4, when performing batch file intelligent recognition, in the batch task, the system automatically matches templates based on text classification, or the user defines temporary extraction fields.

[0017] Preferably, in step S4, fields with low confidence levels are automatically marked as abnormal and then enter the manual review stage.

[0018] Preferably, in step S5, when performing manual verification and log auditing, abnormal fields are displayed centrally and highlighted, allowing auditors to quickly locate the corresponding area in the document and revise the field values.

[0019] Preferably, in step S5, after manual verification and log auditing, the system records the audit log, including the changes in content, time, personnel, and confidence level before and after the modification, forming a complete audit closed loop of "automatic identification - manual verification - process traceability - result traceability".

[0020] The system disclosed in this invention improves the stability and input quality of OCR recognition through multimodal data preprocessing, introduces a semantic understanding mechanism into the recognition process to enhance field location and extraction capabilities, and achieves flexible adaptation to various document types through customizable field extraction templates, thereby improving the automation, accuracy, and scalability of document processing. It is widely applicable to scenarios such as document structured parsing, business information extraction, automated data entry, and intelligent review, and is particularly suitable for the automatic recognition and extraction of key fields in various types and formats of documents such as invoices, contracts, forms, official documents, and reports.

[0021] Compared with existing technical solutions, the present invention has the following beneficial effects: I. To address the issue of low recognition accuracy of existing OCR in scenarios with poor image quality, this invention introduces a multimodal image preprocessing mechanism during the intelligent document parsing stage. This mechanism includes targeted processing methods such as grayscale conversion, binarization, tilt correction, stamp detection, noise suppression, and contrast enhancement. This effectively improves the input quality of documents, reduces interference from factors such as stamp occlusion, uneven lighting, and scanning blur on the recognition results, and significantly improves the stability and recognition accuracy of OCR in complex scenarios.

[0022] Second, to address the problem that existing OCR cannot understand text logic and business semantics and is difficult to achieve accurate field location, this invention introduces a semantic analysis mechanism into the recognition process. Through semantic vector matching and context feature analysis, it can automatically identify templates and locate fields semantically in documents, supporting field extraction in different layouts and positions. This breaks through the limitations of traditional OCR that relies on fixed coordinates or layout mapping, thus achieving high field extraction accuracy and robustness even in documents with inconsistent formats.

[0023] Third, addressing the shortcomings of existing template mechanisms in terms of flexibility and difficulty in quickly adapting to various document types, this invention employs a visual annotation template customization mechanism driven by "layout location + lightweight semantics." This allows users to automatically generate field key-value mapping templates through interactive annotation, reducing the complexity of manually writing rules. Simultaneously, the system supports automatic matching of templates and semantic features, enabling rapid field adaptation and extraction across different document types, significantly improving scalability and maintainability.

[0024] IV. To further ensure the accuracy and traceability of field extraction results, this invention introduces a human-machine collaborative review and log auditing mechanism. For low-confidence fields, the system automatically marks them as abnormal and initiates a manual review process. Through highlighting and one-click revision functions, the cost of manual intervention is significantly reduced. Simultaneously, end-to-end log recording enables field-level operation traceability, providing a reliable basis for quality analysis and algorithm optimization.

[0025] Fifth, in the result output stage, this invention supports multi-format structured data output (including CSV, Excel, JSON, HTML, etc.) and has the ability to customize field mapping and directly connect to databases, enabling efficient integration with external business systems. This technical feature reduces the workload of manual secondary data cleaning and improves the usability of recognition results and data flow efficiency.

[0026] In summary, this invention overcomes the problems of low recognition accuracy and poor adaptability of existing document recognition technologies in complex scenarios through technological innovations in multimodal data preprocessing, semantic understanding, and flexible template mechanisms. It achieves high-precision, highly scalable, and robust automatic extraction and structured output of key fields, demonstrating significant technological advancement and application promotion value. Attached Figure Description

[0027] Figure 1 The system architecture is illustrated; Figure 2 Develop a technology roadmap for the extraction task; Figure 3 A roadmap for data preprocessing technologies; Figure 4 A roadmap for page layout recognition technology; Figure 5 A roadmap for batch intelligent recognition technology; Figure 6 This is a technical roadmap for manual verification. Figure 7 This is a structured output technology roadmap. Detailed Implementation

[0028] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0029] This invention addresses the technical bottlenecks in existing document recognition and information extraction technologies, such as insufficient multimodal data preprocessing, lack of semantic understanding capabilities, and poor template adaptability. It proposes a document key field extraction system based on multimodal OCR recognition and semantic analysis, such as... Figure 1 As shown, it adopts a "four-layer architecture design", namely the data layer, processing layer, service layer and application layer, to realize a closed-loop technology for the entire process from the access of original business documents to intelligent parsing, semantic extraction, review processing and structured output, thereby achieving efficient and automated extraction and processing of multimodal business documents.

[0030] Data layer: Responsible for accessing and storing original business documents, including contracts, payment vouchers, invoices, and bank acceptance bills.

[0031] Processing layer: Responsible for basic parsing such as image preprocessing, layout analysis, OCR recognition, and model invocation.

[0032] Service layer: Includes core service capabilities such as intelligent file parsing, customized annotation templates, batch intelligent recognition, data auditing and optimization, and structured output.

[0033] Application layer: Provides users with a multimodal data intelligent extraction system to achieve integration with external systems and business processes.

[0034] The document key field extraction system based on multimodal OCR recognition and semantic analysis described above is further implemented using the following steps: Step S1, Extract Task Construction: The system constructs field extraction tasks tailored to different document types and business scenarios to drive subsequent batch recognition and parsing processing. It supports two types of task construction: custom tasks and template tasks, with the technical approach as follows: Figure 2 As shown.

[0035] (a) Custom task construction In the task configuration interface, users can directly define the field names, field types, and Prompt semantic prompts to be extracted. The system performs semantic matching and value extraction in the subsequent recognition process based on the defined field information, without the need to create templates in advance.

[0036] (II) Template Task Construction When document types are stable and scenarios are standardized, users can pre-create templates using annotation tools; The template includes information such as the coordinates of the fields, semantic keywords, and field keys; During the task construction process, the system automatically loads the corresponding template and binds the template content to the recognition task.

[0037] (III) Task Configuration and Binding The system generates a unique task identifier and task parameter set for each extraction task, including field definitions, task type, document type identifier, etc. Subsequent identification scheduling and semantic matching are both driven by task configuration, achieving unified scheduling and parallel processing.

[0038] Step S2, data access and preprocessing, the technical route is as follows: Figure 3 As shown: Receive multimodal business documents (PDF, images, scanned copies) through the data acquisition module; The input file is formatted and preprocessed, including grayscale conversion, binarization, stamp detection, and background noise removal. Output a unified intermediate file format for subsequent layout recognition and OCR calls.

[0039] Step S3, layout structure recognition and text extraction, the technical approach is as follows: Figure 4 As shown: Using a lightweight layout structure recognition algorithm, the pre-processed document is segmented to identify structures such as text areas, table areas, and stamp areas. The OCR engine is invoked to recognize text in the located area and output the text content and coordinate information. The recognition results are semantically preprocessed using Prompt prompts.

[0040] Step S4: Batch file intelligent recognition, the technical approach is as follows Figure 5 As shown: In batch tasks, the system automatically matches templates based on text classification, or allows users to define temporary extraction fields; The identification task is processed in parallel through multi-process scheduling; The system combines OCR recognition and semantic matching algorithms to automatically extract key fields. Fields with low confidence levels are automatically marked as abnormal and then enter the manual review stage.

[0041] Step S5, manual verification and log auditing, the technical approach is as follows: Figure 6 As shown: Abnormal fields are displayed centrally and highlighted; Reviewers can quickly locate the corresponding area in the document and revise the field values; The system records audit logs, including changes in content before and after modifications, time, personnel, and confidence level; This forms a complete audit loop of "automatic identification - manual verification - process traceability - result traceability".

[0042] Step S6: Structured output and direct database connection, the technical route is as follows: Figure 7 As shown: The approved data is exported in the target business system format, including CSV, Excel, JSON, HTML, etc. Users can customize field mapping rules; The system supports direct database connection, writing the extracted results to an external database in real time to achieve business linkage.

[0043] The innovations of the above technical solution are mainly reflected in the following aspects: (I) Structural parsing and semantic fusion technology By combining document layout structure information with lightweight semantic analysis results, the template generalization ability is improved and the dependence on fixed layout is reduced. (ii) Dual-mode recognition mechanism of template task + custom task Automatic template matching and flexible temporary extraction are performed in parallel, improving the adaptability and robustness of batch file processing; (III) Human-Machine Collaborative Review Closed-Loop Mechanism By using low-confidence field marking and a visual review interface, we can achieve high accuracy and high efficiency in manual review. (iv) Multi-format structured output and direct system connection It provides a flexible data output and database writing mechanism, enabling the integration of recognition results with other business systems.

Claims

1. A multimodal data intelligent extraction system based on OCR recognition and semantic analysis, characterized in that, The system employs a four-layer architecture consisting of a data layer, a processing layer, a service layer, and an application layer. This architecture enables a closed-loop technology process from raw business document access to intelligent parsing, semantic extraction, review processing, and structured output. It achieves efficient and automated extraction and processing of multimodal business documents. Specifically: the data layer handles the access and storage of raw business documents; the processing layer handles basic parsing including image preprocessing, layout analysis, OCR recognition, and model invocation; the service layer provides core services including intelligent document parsing, customized annotation templates, batch intelligent recognition, data review optimization, and structured output; and the application layer provides users with a multimodal data intelligent extraction system, enabling integration with external systems and business workflows.

2. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 1, characterized in that, The original business documents include contracts, payment vouchers, invoices, and banker's acceptances.

3. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 1, characterized in that, The multimodal data intelligent extraction system is implemented using the following steps: Step S1: Construct field extraction tasks for different document types and business scenarios to drive subsequent batch recognition and parsing processing. The system generates a unique task identifier and task parameter set for each extraction task. Subsequent recognition scheduling and semantic matching are driven by task configuration, achieving unified scheduling and parallel processing; Step S2: After receiving multimodal business documents through the data acquisition module, the input files are format-standardized and preprocessed, outputting a unified intermediate file format for subsequent layout recognition and OCR use; Step S3: Utilize a lightweight layout structure recognition algorithm to process the preprocessed data... The document is segmented into regions, and the OCR engine is called to recognize the text in the located regions, outputting the text content and coordinate information. Prompt words are used to perform semantic preprocessing on the recognition results; Step S4: Batch file intelligent recognition. The recognition task is processed in parallel through multi-process scheduling, and the system combines OCR recognition and semantic matching algorithms to automatically extract key fields; Step S5: Manual verification and log auditing; Step S6: The approved data is exported according to the target business system format. Users can customize field mapping rules, and the system supports direct database connection, writing the extracted results to the external database in real time to achieve business linkage.

4. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 3, characterized in that, In step S1, two types of extraction task construction methods are supported: custom task construction and template task construction. To complete the custom task construction, the user can directly define the field name, field type and Prompt semantic prompt in the task configuration interface. The system will then perform semantic matching and value extraction in the subsequent recognition process based on the defined field information, without the need to create a template in advance. In template task construction, when the document type is stable and the scene is standardized, the user can pre-create the template using the annotation tool. The template includes the coordinate position of the field, semantic keywords, and field key. During the template task construction process, the system automatically loads the corresponding template and binds the template content to the recognition task.

5. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 3, characterized in that, In step S1, the task parameters include field definitions, task type, and document type identifier.

6. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 3, characterized in that, In step S2, the preprocessing includes grayscale conversion, binarization, stamp detection, and background noise removal.

7. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 3, characterized in that, In step S4, when performing batch file intelligent recognition, the system automatically matches templates based on text classification in the batch task, or allows users to define temporary extraction fields.

8. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 3, characterized in that, In step S4, fields with low confidence levels are automatically marked as abnormal and then enter the manual review stage.

9. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 3, characterized in that, In step S5, during manual verification and log auditing, abnormal fields are displayed and highlighted in a centralized manner, allowing auditors to quickly locate the corresponding areas in the document and revise the field values.

10. The multimodal data intelligent extraction system based on OCR recognition and semantic analysis as described in claim 3, characterized in that, In step S5, after manual verification and log auditing, the system records the audit log, including the changes in content, time, personnel, and confidence level before and after the modification, forming a complete audit closed loop of "automatic identification - manual verification - process traceability - result traceability".

Images