Document data engine method and system

Abstract

The invention relates to the related field of document data engines, in particular to a document data engine method and system, which realizes high-robustness field extraction capability by combining a template engine with an AI model, and adapts to multiple types of unstructured documents; field splitting / combination is intelligent, and manual intervention is reduced. Automatic mapping of standard fields and alignment of platform fields are achieved, the adaptation relation between an industry field standard library and multi-system fields is constructed, and the fields can be directly pushed to multiple service systems from original samples; field data quality and consistency guarantee capability are improved, and output fields are more accurate due to field-level difference comparison and format restoration functions; the AI completion and fuzzy recognition mechanism solves the problem that fields are not covered or uncertain in the traditional recognition technology. The system has four characteristics of standardization, rule driving, model assistance and dynamic updating, and the manual operation intensity and the business risk are remarkably reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of document data engine, and in particular to a document data engine method and system. BACKGROUND

[0002] International logistics has various links and many participants, and import and export business involves massive file transmission and maintenance. In the four core links of booking, booking, material cutting and VGM cutting, the customer service, booking and document personnel of the forwarding agent enterprise highly rely on manual operation: copying and pasting on the sample, extracting, splitting, merging and converting information according to experience to ensure accurate entry of information in the forwarding agent system and the ship official website. This mode consumes a lot of manpower and significantly increases the cost of the enterprise.

[0003] The prior art can usually only extract the original information of the sample. The industry standardization processing of the information and the effective linkage with the ship official website and the forwarding agent system have not been effectively broken through. SUMMARY

[0004] The present application aims to provide a document data engine method and system to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solution: a document data engine method, comprising a forwarding logistics system, the method comprising the following steps, S1, receiving uploaded document samples, archiving and storing and recording meta information with a unique identity; S2, analyzing the meta information, extracting the text information to construct a structured graph; S3, standardizing the text information of the structured graph based on a preset field alias mapping library and a regular rule library; S4, optimizing the standardized text information; S5, mapping the optimized text information with the international logistics system (platform field) one by one through a mapping engine; S6, using a large language model to infer and complete the text information problem field identified one by one in step S5; S7, outputting the data engine result based on the completed text information.

[0006] As a preferred, the file format type of the document sample in step S1 includes picture, PDF, Word, Excel, email, and the meta information includes the file name of the document sample, customer information and document type.

[0007] As a preferred, the text information in step S2 includes text content, field position, field paragraph, table row and column, and font style.

[0008] Preferably, the text information optimization in step S4 includes the following sub-steps: S41. Decompose the fields to be split in the text information based on the named entity recognition model; S42. Merge the fields to be concatenated in the text information based on the rule concatenation tool; S43. Standardize the format of text information based on regular expression rule base; S44. Correcting erroneous expressions of text information based on pre-trained language models.

[0009] Preferably, step S6 uses a confidence scoring mechanism to mark fields that are identified as being below a threshold.

[0010] Preferably, the data engine result in step S7 is in JSON format.

[0011] Sample parsing module: Based on a multi-engine OCR fusion algorithm, it extracts the original content of the document sample and parses the metadata; Field recognition and standardization module: Based on a preset field alias library and regular expression matching rule library, it realizes the unified normalization of field names in the sample and extracts structured JSON data from unstructured data; Standardized Fields Module: Standardizes structured JSON data based on a sequence labeling model; Field mapping and multi-system field alignment module: performs field conversion processing, mapping standard fields to the fields required by each user; AI-assisted judgment module: Based on a large language model, it trains on document samples to improve the ability to reason about and complete missing or ambiguous fields. Strategy Execution and Log Backtracking Module: Links with the above modules to record relevant input and output data for backtracking.

[0012] Compared with the prior art, the present invention provides a document data engine method and system, which has the following beneficial effects: 1. Achieve highly robust field extraction capabilities by combining a template engine with an AI model, adapting to various types of unstructured documents; intelligent field splitting / combination reduces manual intervention and improves field recognition accuracy and automated processing capabilities.

[0013] 2. Build an industry-standard field library and adapt it to fields in multiple systems, so that fields can be directly pushed from the original sample to multiple business systems; support hot updates of new fields, aliases, and mapping rules, and realize automatic mapping of standard fields and alignment with platform fields.

[0014] 3. Field-level difference comparison and format repair functions make the output fields more accurate, improving the quality and consistency of field data; AI completion and fuzzy recognition mechanisms solve the problem of uncovered or uncertain fields in traditional recognition technologies.

[0015] 4. It possesses four major characteristics: standardization, rule-driven, model-assisted, and dynamic updates, which enhances the system's scalability and industry-wide applicability; it can be widely applied to the document processing needs of different customers, shipping companies, and platforms.

[0016] 5. Automatic extraction of multiple fields, semantic-level data structure reconstruction, and verification and comparison capabilities significantly reduce manual judgment in the operation process, thereby significantly reducing the intensity of manual operations and business risks; ensuring data compliance and avoiding risks such as incorrect filling, omissions, and format errors.

[0017] The features and advantages of the present invention will be described in detail through embodiments and in conjunction with the accompanying drawings. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the steps in an embodiment of the present invention. Detailed Implementation The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 This invention provides a document data engine method and system, which mainly includes the following modules: 1. Sample parsing module: It employs a multi-engine OCR fusion algorithm, combined with a layout analysis engine, for document paragraph segmentation, table border recognition, and text block positioning; It supports extracting the original content of multiple file formats (images, PDFs, Word, Excel, email bodies) using tools such as Apache POI, PDFBox, and LibreOffice API, and combines OCR to correct incomplete text or extract parts embedded in images; Generate a key-value tree structure based on location coordinates, text content, and hierarchical structure to provide contextual support for subsequent field recognition.

[0020] 2. Field Recognition and Standardization Module: Establish a field alias library and a regular expression matching rule library, and achieve unified normalization of field names in different samples through hash mapping and regular expressions (e.g., Shipper / Shipper / Sender → Shipper_Name). Field mappings are pre-established and stored in databases; the fields are typically standard fields and a list of mapped fields.

[0021] By combining pre-trained models such as BERT / ERNIE to encode the field context, semantic similarity matching between the field and the standard definition is performed; For numeric, date, and currency fields, a dual verification mechanism using type checking and regular expressions is employed to ensure standardization accuracy; for example: email:@ 1 testcom This string matches the string rules based on its type. However, it doesn't conform to the email address format rules. Regular expressions can be used to determine if it's an email address. It provides a dynamic expansion mechanism, manages aliases and mapping rules through a configuration center, and supports online updates.

[0022] 3. Standardized Fields Module: Using sequence labeling-based word segmentation and entity recognition, complex fields such as contact information and addresses are accurately decomposed; By combining syntactic dependency analysis to determine the syntactic logic within a field, semantic hierarchical segmentation can be achieved; Use regular expressions combined with rule trees (such as extraction rules for phone numbers and email addresses) for rapid splitting; Field concatenation uses template concatenation strategies and rules to drive the concatenator, such as concatenating "province / city / district" into a complete address, and supports conversion between Chinese and English / multi-language formats.

[0023] Format correction utilizes a rule engine with built-in rules for converting and standardizing various data formats, including dates, amounts, and phone numbers. Version comparison mechanism: The system maintains a historical version chain for fields and uses a difference analysis algorithm to generate difference reports and prompts; it also supports custom error tolerance configurations, such as numerical range tolerance and time error interval.

[0024] 4. Field mapping and multi-system field alignment module: A standard industry field library and a system field mapping database are built, and the knowledge graph automatically completes the mapping; the structure is as follows, and what needs to be converted are the field names: Standard field {field name, format}; The fields for shipping company A are defined as follows: {Field Name, Format}. The fields for shipping company B are defined as follows: {Field Name, Format}. This allows the standard fields to be converted into the fields required by shipping company A.

[0025] For complex many-to-many mappings, use a mapping configuration DSL (domain-specific language) to flexibly configure the mapping relationship from multiple sources to target fields; for many-to-many fields: the three fields A(a), B(b), and C(c) and their values ​​are reorganized into X(a+b) and Y(b+c).

[0026] Provides a bidirectional mapping converter to ensure field consistency from document samples to business systems and reverse verification; Integrate open APIs and middleware services (such as ESB and Kafka) to achieve cross-system data synchronization and mapping.

[0027] 5. AI-assisted judgment module: Fine-tune's large language model is specifically trained for the freight forwarding documentation domain to improve the contextual reasoning and completion capabilities for missing and ambiguous fields; The system uses a model confidence scoring mechanism to determine whether a field needs manual review. Configure a human-machine collaboration mechanism so that low-confidence models are automatically transferred to a human review platform. After passing the review, feedback is given to the model to continuously improve its generalization. AI judgment results are recorded in the audit log and feedback database for continuous optimization of model inference accuracy.

[0028] 6. Strategy Execution and Log Backtracking Module: Logs are recorded throughout the entire process, and distributed tracing and auditing are performed using ELK (Elasticsearch + Logstash + Kibana). It supports version management of strategies and models. Version control adopts a Git-like change management mechanism, which allows for rollback or comparison of the effects of different versions at any time. The rules and models support a hot-reload mechanism, dynamically publishing updates through the Nacos configuration center without requiring a service restart.

[0029] The document data engine includes the following process steps: Step 1: Receive the document sample uploaded by the user. It provides multiple methods for receiving document files, including web interface, API interface, and SFTP upload; Supported file formats include: PDF, Word, Excel, email body, etc. After uploading, the file is archived with a unique ID, stored in Object Storage Service (OSS), and metadata (file name, client, document type) is recorded.

[0030] Step 2: Parse the text information of the uploaded document sample to generate a structured graph. For image-based PDFs, an OCR layout parsing model is used to extract text and layout information. For text documents (such as Word and Excel), Apache POI can be used to extract the text and table structure directly. The parsed output includes: text content, field location, paragraph, table rows and columns, font style, etc., and generates a structured representation of the intermediate layer (such as a JSON-based key-value pair structure + hierarchical coordinate information).

[0031] Step 3: Standardize the fields of the generated structured graph. A field alias mapping library and a regular expression rule library are constructed for field normalization (e.g., "Shipper" and "Shipper" are unified as "shipper_name"). The BERT model is used to calculate the semantic similarity of fields to help solve the fuzzy mapping problem in alias normalization. The format of data such as numerical values, dates, and amounts is unified and standardized into a unified format within the system.

[0032] Step 4: Perform field-level splitting / combining processing on the text information. For composite fields such as "Contact Information" and "Shipper Address," the Named Entity Recognition (NER) model is used to break down entities, and regular expressions and template rules are used to accurately extract information such as phone numbers, email addresses, and postal codes. For fields that need to be concatenated (such as province, city, and district combined into a "complete address"), a rule-based concatenation tool and field order logic are used to achieve merging.

[0033] Step 5: Convert the field formats and perform semantic correction on the text information. The format of dates, amounts, and phone numbers is standardized by using a regular expression rule base and a type detection module. Pre-trained language models and domain knowledge bases (such as port alias databases and UN number databases) are used to correct semantic ambiguities or incorrect expressions (such as “Los Angeles (United States)” → “USLAX”).

[0034] Step 6: Map to target platform field. Based on industry field standard libraries and field mapping configurations, the mapping engine automatically maps standardized fields in text information to target platform fields one-to-one. It supports multi-platform configuration (such as freight forwarding systems and shipping company systems), with each platform defining different field mappings and format conversions. It has a two-way mapping mechanism and supports reverse verification and synchronization.

[0035] Step 7: Use AI completion and field comparison to verify the text information after one-to-one correspondence. For missing or uncertain fields, a domain-fine-tuned large language model is used to infer and complete the fields based on context. A confidence scoring mechanism is used to mark fields below a certain threshold as "requiring manual review".

[0036] Field comparison: Compare with historical entrustment documents, customer master data, historical order information, etc., using a dual comparison of hash + vector similarity to check for synonyms, numerical differences, date errors, etc., and provide difference prompts.

[0037] Step 8: Based on the completed and validated text information, output structured JSON or automatically push the output. Implementation methods: Supports JSON standard format output, unified API integration, and connection to freight forwarding systems, shipping company websites, customer middleware, etc. Push via RESTful API, WebHook, message queue (such as Kafka), etc., while generating operation logs and comparison reports for auditing and tracking.

[0038] As a feasible embodiment of the present invention For example, processing the field "Customer Contact Person: zhangsan, Phone: +86-13800000000, Email: zhangsan@abc.com" in the "Entrustment Letter" field. It is split into three fields (contact person, phone number, and email); mapped to standard fields such as contact_name, contact_phone, and contact_email; the format automatically recognizes the country code and verifies the validity of the email; and pushed to the customer module interface field in the freight forwarding system.

[0039] Implementation Method 1: Structured Extraction and Standard Mapping of Shipper Fields The original input sample consists of a PDF file uploaded by the user, containing information such as "Shipper: ABC Logistics Ltd.", "Container Type: 40HQ", and "Port of Discharge: Los Angeles".

[0040] Sample parsing module execution The system first calls the embedded OCR engine to convert the PDF text into a recognizable structure; such as { "fields": [ { "field_text": "Shipper: ABC Logistics Ltd.", "field_type": "Shipper_Name", "position": { "x": 100, "y": 250, "width": 300, "height": 20, "page": 1 }, "font_info": { "font_name": "Arial", "font_size": 12, "bold": false }, "confidence": 0.98, "source_page": 1, "context": "Shipper: ABC Logistics Ltd.\nConsignee: XYZ ImportInc.", "block_id": "block_001", "hierarchical_path": "Section 1>Paragraph 2", "parent_field": null, "extracted_at": "2025-07-13T16:20:00Z" }, { "field_text": "Address: 123 Export Street, Shanghai, China", "field_type": "Shipper_Address", "position": { "x": 100, "y": 275, "width": 350, "height": 20, "page": 1 }, "font_info": { "font_name": "Arial", "font_size": 12, "bold": false }, "confidence": 0.95, "source_page": 1, "context": "Shipper: ABC Logistics Ltd.\nAddress: 123 ExportStreet, Shanghai, China", "block_id": "block_001", "hierarchical_path": "Section 1>Paragraph 2", "parent_field": "Shipper_Name", "extracted_at": "2025-07-13T16:20:01Z" } ] } At the same time, it combines layout recognition components (such as table borders and paragraph blocks) to form a preliminary set of candidate fields.

[0041] Field standardization recognition The system uses a built-in field naming alias library to uniformly map synonymous fields such as "Shipper", "Shipper Name", and "Sender" to "Shipper Name"; and matches the "40HQ" field to the system's standard field "Box Type Code", mapping it to "40-foot High Container".

[0042] Field structure conversion and repair 1. Rule validation phase, Based on the field type (phone, date, amount, address, etc.), the built-in rule engine is first executed to perform format and content validity checks: Using regular expressions, dictionary mappings, and hard rules such as numerical values / length, when validation fails, the error code and reason are recorded, and an AI model is triggered for assistance.

[0043] Example: Phone number format validation failed → ERROR_PHONE_FORMAT; invalid or ambiguous date → ERROR_DATE_FORMAT. In business systems, the code "ERROR_PHONE_FORMAT" is typically passed. For example, if a field's description code is "ERROR_PHONE_FORMAT", it means "phone number format validation failed".

[0044] 2. AI model correction hints and inference completion. For fields that fail validation, the system automatically sends the original content, type, and context of the field into the large model using a preset Prompt prompt template.

[0045] Prompt template example, such as field type: ${field type}, field value: ${original value}, context: ${context}. Please help me correct the content of this field and return the standardized result.

[0046] 3. Secondary verification of rules Use the rules engine to verify the results returned by AI again to confirm whether the corrected results are compliant and complete. If passed: Update the field value. If not passed: Mark as a mandatory item for manual review. For example: Field A originally had no value, and the result of AI adding the value "20250101" needs to be verified again to see if it meets the specifications.

[0047] If there is an error in the field content, such as "O" in "+86138000O000" being a letter, the system automatically identifies and replaces it with the number "0"; convert "September 28, 2023" to "2023-09-28" to complete unified formatting.

[0048] Output structured results: The output structured result data can be pushed to the freight forwarder system through the HTTP API or written to the database; and automatically align and verify fields with the corresponding shipping company field mapping structure.

[0049] Implementation method 2: Field splitting and combination example (contact information) Input field content: css CopyEdit Contact: Tom / Phone: +86-13800000000 / Email: tom@abc.com System processing process: The system identifies that this line belongs to a composite field; uses regular expressions to extract and split it into three parts: contact_name: Tom, contact_phone: +86-13800000000, contact_email: tom@abc.com, Output structure format: Implementation method 3: Repair of non-standard field content and AI completion Input scenario: The content of the "destination port" field is identified as "Los Angeles (USA)", but the port code "USLAX" needs to be matched in the business system. Processing logic: The system identifies "Los Angeles" → "Los Angeles" based on the port alias library; Obtain the corresponding UN / LOCODE from the port standard library; The corrected result is: "USLAX".

[0050] If a field is missing: The AI ​​module determines the possible port of destination as "Los Angeles" based on context such as "California, USA" in the "Consignee Address" and automatically completes "Port of Discharge".

[0051] Implementation Method 4: System Integration. The system supports outputting JSON, XML, Form-Data, and other formats according to customer requirements to meet the needs of different system integrations. For example, for API interface calls, a RESTful interface is provided: POST / api / v1 / doc / parse, Body: { file, doc_type, customer_id}, Response: standard field structure + comparison logs, integrating with freight forwarding systems. Structured fields are pushed to the freight forwarding system, and the fields in the shipping company's booking system are automatically populated through mapping rules, enabling end-to-end automated order processing.

[0052] Implementation Method 5: The rules feature dynamic configuration and hot update capability. All field mapping, splitting, combination, and format conversion rules support user-defined configuration and are stored in a versioned rule library.

[0053] The hot update mechanism allows administrators to upload rule files when new fields are added or shipping standards change, and the system can be updated without restarting.

[0054] Through the above-mentioned multiple implementation methods, this system realizes an intelligent data engine for the entire process from original document samples to structured business fields and then to business system integration, which significantly improves the efficiency and accuracy of integration between cross-document, multi-platform, and multi-business systems.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A document data engine method, including a freight forwarding logistics system, the method comprising the following steps, S1. Receive uploaded document samples, archive and store them with a unique identifier, and record metadata. S2. Parse the metadata, extract text information, and construct a structured graph; S3. Standardization of text information in structured graphs based on a preset field alias mapping library and a regular expression rule library; S4. Optimize the standardized text information; S5. The optimized text information is mapped one-to-one with the international logistics system through the mapping engine; S6. Use a large language model to infer and complete the text information problem fields that are identified one by one in step S5. S7. Output data engine results based on text information completion.

2. The document data engine method according to claim 1, characterized in that, In step S1, the file format types of the document samples include images, PDFs, Word documents, Excel files, and emails. The metadata includes the file name of the document sample, customer information, and document type.

3. The document data engine method according to claim 1, characterized in that, The text information in step S2 includes text content, field position, field paragraph, table rows and columns, and font style.

4. The document data engine method according to claim 1, characterized in that, The optimization of text information in step S4 includes the following sub-steps: S41. Decompose the fields to be split in the text information based on the named entity recognition model; S42. Merge the fields to be concatenated in the text information based on the rule concatenation tool; S43. Standardize the format of text information based on regular expression rule base; S44. Correcting erroneous expressions of text information based on pre-trained language models.

5. The document data engine method according to claim 1, characterized in that, In step S6, a confidence scoring mechanism is used to mark fields that are identified as being below a threshold.

6. The document data engine method according to claim 1, characterized in that, In step S7, the data engine result is in JSON format.

7. A document data engine system, characterized in that, The system is used to implement the document data engine method according to any one of claims 1 to 6, and the system includes, Sample parsing module: Based on a multi-engine OCR fusion algorithm, it extracts the original content of the document sample and parses the metadata; Field recognition and standardization module: Based on a preset field alias library and regular expression matching rule library, it realizes the unified normalization of field names in the sample and extracts structured JSON data from unstructured data; Standardized Fields Module: Standardizes structured JSON data based on a sequence labeling model; Field mapping and multi-system field alignment module: performs field conversion processing, mapping standard fields to the fields required by each user; AI-assisted judgment module: Based on a large language model, it trains on document samples to improve the ability to reason about and complete missing or ambiguous fields. Strategy Execution and Log Backtracking Module: This module works in conjunction with the above modules to record relevant input and output data for backtracking purposes.

Images