Fast recognition method for archive entry based on multi-mode self-adaptive OCR (optical character recognition)

Abstract

The invention relates to the technical field of document image analysis and optical character recognition, and discloses a quick recognition method for archive entry based on multi-modal self-adaptive OCR (optical character recognition), which comprises the following steps of: S1, receiving an image file of an unstructured archive; s2, an improved multi-mode layout analysis model is adopted, layout analysis is conducted on the image file, and text semantic features are fused in the improved multi-mode layout analysis model. An improved LayoutLMv3 model is adopted, particularly, a domain self-adaptive visual adapter module is introduced, and the domain self-adaptive visual adapter module carries out pre-training on specific visual elements (such as lithologic symbols, handwritten textures and table thin lines) in the hydrogeological field. According to the method, the model is not a'universal 'model any more, but a'field-specific' model, the complex mixed layout containing professional legend and handwritten annotation can be accurately segmented, and the error rate of layout analysis is greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to document image analysis and optical character recognition technology, focusing on solving the problems of automatic text extraction and layout analysis of unstructured archives. BACKGROUND

[0002] With the implementation of policies such as the "Groundwater Management Regulations" and the "National Geological Disaster Prevention and Control "Fourteenth Five-Year Plan", the coal mine water prevention and control work is accelerating the transformation to digitalization and intelligentization. As the core carrier for the integration, sharing and utilization of hydrogeological data of coal mines nationwide, the key module of the water prevention and control cloud platform, the hydrogeological digital data library, includes massive core data such as drilling data, geological profile maps, and hydrological observation records. However, such data is mostly stored in unstructured forms such as scanned copies and pictures, and it is urgently needed to convert them into editable and searchable structured data through optical character recognition (OCR) technology to meet the borrowing, analysis and secondary utilization needs and fully release the application value of the data.

[0003] OCR technology has now entered the deep learning driven stage, and multi-modal OCR technology based on LayoutLM series models can integrate text and layout information, achieving remarkable results in the fields of archives and industry.

[0004] For example, the household archives automatic registration system built based on Baidu PaddleOCR reduces the error rate from 18% to below 3% and improves efficiency by 300%; the intelligent archives open audit system realizes seamless connection between "recognition and storage"; the "OCR + multi-modal information extraction" archives quality inspection model balances accuracy and cost.

[0005] Although the generalizability of OCR technology is continuously improving, its application in professional fields such as hydrogeology still has obvious shortcomings.

[0006] When processing hydrogeological data, traditional OCR and existing deep learning models mainly face the following four key difficulties:

[0007] The same data may have single-column, double-column or even multi-column formats (such as report text, journal excerpts, and data compilation), and general layout analysis algorithms are prone to misjudge the boundaries of text areas, leading to content cutting errors.

[0008] Hydrogeological terms are dense (such as "Ordovician limestone aquifer" and "fractured zone"), which differ greatly from ordinary words, and traditional segmentation models are prone to error splitting or misidentification; numerical units in tables (such as "m 3 / h" and "mD") are often missed or misjudged.

[0009] Some old drilling data scanned before 2000 have problems such as faded handwriting, page tilt (tilt angle up to 15°), and black edge stains, which seriously affect the accuracy of text detection and character recognition.

[0010] Information such as "borehole number - water level" in geological profile maps needs to be combined with spatial location relationships for identification. Traditional OCR can only extract isolated text and cannot establish semantic associations.

[0011] These problems result in existing OCR systems achieving an average recognition accuracy of only 83.5% and a table parsing accuracy of 82.1% in the digitization of hydrogeological data. The processing efficiency is 12 pages per minute, which is far below business requirements (more than 500 pages per day), and the data utilization rate is less than 30%.

[0012] Existing technologies cannot simultaneously address the professionalism, complexity, and quality variations of hydrogeological data, resulting in high digitization costs and low efficiency. Summary of the Invention

[0013] This invention provides a fast document entry and recognition method based on multimodal adaptive OCR, which solves the problems mentioned in the background art.

[0014] This invention provides the following technical solution: a method for rapid document entry and recognition based on multimodal adaptive OCR, comprising the following steps:

[0015] Step S1: Receive the image file of the unstructured archive;

[0016] Step S2: An improved multimodal layout analysis model is used to perform layout analysis on the image file. The improved multimodal layout analysis model integrates text semantic features, two-dimensional position features, and domain-enhanced visual features to segment the image file into one or more logical regions.

[0017] Step S3: Call the OCR engine to perform text recognition on the one or more logical regions and output the original recognition data;

[0018] Step S4: Based on the domain knowledge base, perform data standardization processing on the original identification data;

[0019] Step S5: Verify the results of the standardized data, and optimize the improved multimodal layout analysis model based on the verification results using closed-loop feedback.

[0020] Preferably, the improved multimodal layout analysis model is constructed by connecting a domain adaptive visual adapter module in series on a basic multimodal model; the domain adaptive visual adapter module is configured to perform domain feature enhancement on the preliminary visual features of the image file before the basic multimodal model fuses features, so as to output the domain-enhanced visual features.

[0021] Preferably, the domain-adaptive visual adapter module is a lightweight convolutional neural network pre-trained for specific visual elements in a specific professional domain.

[0022] Preferably, the domain knowledge base stores professional terms, standardization rules, and data verification rules for the data standardization processing and the result verification.

[0023] Preferably, the standardization rules and data verification rules include: a list of allowed values ​​for technical terms; and the type, unit, and / or reasonable range of numerical data.

[0024] Preferably, step S4 includes: performing terminology correction and / or unit completion on the original identification data based on the domain knowledge base.

[0025] Preferably, step S5 includes: automatically screening the standardized data based on the data verification rules in the domain knowledge base to mark outliers; highlighting the outliers in the visualization interface to receive manual correction; and closed-loop feedback optimization includes: collecting the results of the manual correction and using the results as semi-supervised learning samples for parameter iterative optimization of the improved multimodal layout analysis model.

[0026] Preferably, the text semantic features, two-dimensional positional features, and domain-enhanced visual features are fused by summing through an embedding layer.

[0027] A rapid document entry and recognition system based on multimodal adaptive OCR includes:

[0028] The input receiving module is configured to receive image files from unstructured archives;

[0029] The multimodal layout analysis module is configured to perform layout analysis on the image file using an improved multimodal layout analysis model;

[0030] The OCR recognition module is configured to invoke the OCR engine for the one or more logical regions;

[0031] The data standardization module is configured to perform data standardization processing on the original identification data based on a domain knowledge base.

[0032] The result verification and feedback module is configured to verify the standardized data and perform closed-loop feedback optimization on the improved multimodal layout analysis model based on the verification results.

[0033] Preferably, the multimodal layout analysis module includes a basic multimodal model and a domain-adaptive visual adapter module connected in series between the visual backbone network and the fusion layer of the basic multimodal model. The domain-adaptive visual adapter module is configured to perform domain feature enhancement on the preliminary visual features extracted by the basic multimodal model.

[0034] The present invention has the following beneficial effects:

[0035] 1. This document entry and recognition method based on multimodal adaptive OCR employs an "improved LayoutLMv3 model," particularly through the introduction of a domain-adaptive visual adapter module. This module is pre-trained for specific visual elements in the hydrogeological field (such as lithological symbols, handwritten textures, and table lines). This transforms the model from a "general" model into a "domain-specific" model, capable of accurately segmenting complex mixed layouts containing professional illustrations and handwritten annotations, significantly reducing the error rate of layout analysis.

[0036] 2. This rapid document entry and recognition method based on multimodal adaptive OCR incorporates a "hydrogeological professional knowledge base" through a data standardization module. After OCR recognition, the data standardization module immediately performs terminology verification and unit completion on the results. This solves the problem of general OCR models "not recognizing" professional terms, ensuring the professional accuracy of the recognition results.

[0037] 3. This multimodal adaptive OCR-based rapid document entry and recognition method automatically verifies the recognition results using rules in a knowledge base through a verification mechanism. It moves beyond "blind" data entry and proactively marks "outliers," achieving closed-loop control of data quality.

[0038] 4. This document entry and recognition method based on multimodal adaptive OCR is a "one-time" recognition tool in existing technologies. This invention, through a feedback mechanism, automatically feeds the results of manual proofreading back to the model training library for iterative parameter optimization of the "improved LayoutLMv3 model." This makes the method of this invention "smarter with use," continuously adapting to new handwriting styles or blurred handwriting, achieving adaptive evolution of the model. Attached Figure Description

[0039] Figure 1 This is a technical roadmap for the system of the present invention;

[0040] Figure 2 This is a schematic diagram illustrating the feature fusion principle of the multimodal layout analysis module of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Please see Figure 1 - Figure 2 ,

[0043] This invention provides a fast document entry and recognition method based on multimodal adaptive OCR, comprising the following steps:

[0044] Step S1: The receiving input module receives image files of unstructured archives; the receiving input module provides one or more interfaces, such as web upload or API call, for receiving unstructured archive image files.

[0045] In the image denoising and enhancement stage, a Gaussian filtering algorithm (3×3 filter kernel) is used to smooth the random noise in the scanned document. The noise intensity of the filtered image is calculated by formula (1), and the noise reduction reaches 40% to 50%. At the same time, the image gray distribution is adjusted by combining histogram equalization technology, which improves the contrast of the blurred text by 20% to 30%, ensuring that the improved LayoutLMv3 model can effectively capture the edge features of the text.

[0046]

[0047] Where I(i,j) represents the pixel value of the original image. represents the pixel values ​​of the filtered image, and M×N represents the total number of pixels in the image.

[0048] In the tilt correction stage, the tilt direction of the text line is detected based on the Hough transform principle, and the tilt angle is automatically calculated (supporting accurate correction within the range of -15° to 15°). The tilt angle is solved by formula (2), and the tilt angle error after correction is <0.5°. The tilted text is adjusted to be horizontally arranged by the bilinear interpolation algorithm to avoid misjudgment of the position features of the LayoutLMv3 model caused by tilt.

[0049] θ = argfmax θ∈[-15°，15°] ∑ (x,y)∈L ρ(x,y,θ) (2)

[0050] Where L is the set of pixels in the text line, and ρ(x,y,θ)=xcosθ+ysinθ is the polar radius of the Hough transform.

[0051] In the black border trimming stage, the effective content area of ​​the page is identified by the Canny edge detection algorithm, and the edge black border and redundant blank area generated during the scanning process are automatically trimmed (trimming accuracy ±1mm). The black border trimming rate is calculated by formula (3), and the trimming rate reaches more than 98%, which reduces the interference of irrelevant areas on the layoutLMv3 model layout analysis and improves the processing efficiency.

[0052]

[0053] Step S2: An improved multimodal layout analysis model is used to perform layout analysis on the image file. The improved multimodal layout analysis model integrates text semantic features, two-dimensional position features, and visual features enhanced by the domain to segment the image file into one or more logical regions.

[0054] The multimodal layout analysis module does not use the standard LayoutLMv3 model, but instead uses an "improved LayoutLMv3 model" to adapt to the specific visual characteristics of the hydrogeological field.

[0055] The main areas for improvement include two points:

[0056] First, sample expansion and fine-tuning: 5,000 pages of labeled samples covering three core types of hydrogeological data were collected. The LayoutLMv3 pre-trained model was fine-tuned in the domain to optimize the model's ability to perceive the layout features of hydrogeological data. The fine-tuning process used the AdamW optimizer with a learning rate of 2e-5 and 10 iterations, which improved the accuracy of layout type judgment to 97.2%.

[0057] Second, feature fusion optimization: based on the original "text semantic features + two-dimensional positional features" of LayoutLMv3, "visual texture features" (image texture extracted through ResNet50) are added to construct a "semantic-positional-visual" three-modal input as shown in formula (4), which solves the problem of insufficient robustness of traditional LayoutLMv3 in analyzing low-quality data (faded, stained) layouts, reducing the misjudgment rate of multi-column text from 12.5% ​​to 3.8%. The principle of multimodal feature fusion is explained in [reference needed]. Figure 2 .

[0058] F fusion =αF semantic +βF position +γF texture (4)

[0059] Where F semantic For text semantic features, F position For two-dimensional positional features, F texture The visual texture features are represented by α, β, and γ, which are the feature weights (set to 0.4, 0.3, and 0.3 after cross-validation).

[0060] In the layout type determination stage, the improved LayoutLMv3 model integrates text line density (number of text lines per square centimeter), regional distribution features (percentage of text block bounding rectangles), and visual texture features. Through the Transformer encoder, it automatically learns the layout pattern of hydrogeological data, distinguishes different formats such as single-column and multi-column, horizontal and vertical layouts, and provides a basis for the selection of subsequent differentiated recognition strategies.

[0061] During the region segmentation stage, the LayoutLMv3 model outputs the bounding box coordinates of the text area, table area, graphic area and annotation area through a fully connected layer (IOU threshold set to 0.7). The text area (such as the main text of the exploration report) and the table area (such as the borehole data statistics table) are the core recognition objects. The graphic area (such as the geological profile) only extracts the text annotation information (such as borehole number and water level elevation). The annotation area (such as handwritten modification opinions) is marked separately to avoid interfering with the recognition of core data.

[0062] During the regional priority ranking stage, computing resources are allocated according to the priority of "table area > text area > annotation area". Priority is given to processing areas with complex structures and high data value, such as tables, to ensure the identification accuracy and processing efficiency of key hydrogeological data (such as aquifer parameters and water level observations).

[0063] In the original LayoutLMv3 model, its input X is a fusion of three parts: text embedding, visual embedding, and layout embedding (positional embedding). Let the text term T be... i The corresponding text embedding is Etext(Ti), and its corresponding two-dimensional location bounding box B. i The layout is embedded as E layout (B i ), its corresponding image block T i The visual embedding is E visual (I i ).

[0064] The input embedding Xi of the i-th token can be represented as:

[0065] X i =E text (T i )+E layout (B i )+E visual (I i (5)

[0066] Among them, the layout embeds E layout (B iTypically, the coordinates (x1, y1, x2, y2) of the top-left and bottom-right corners of the bounding box are obtained by embedding them separately using a multilayer perceptron (MLP) and then summing them together.

[0067] E layout (B i ) = MLP x1 (x1)+MLP y1 (y1)+MLP x2 (x2)+MLP y2 (y2) (6)

[0068] The LayoutLMv3 model performs deep context fusion on the input embedding sequence {x1} through a multi-layer Transformer encoder:

[0069] {H i} = TransformerEncoder({X i}) (7)

[0070] The pre-training tasks for the LayoutLMv3 model include:

[0071] Masked Language Modeling (MLM): Predicts masked text terms. Its loss function... It can be represented as:

[0072]

[0073] (in (A set of masked text lexical indexes)

[0074] Masked Image Modeling (MIM): Predicts masked visual feature patches. Its loss function is... It can be represented as:

[0075]

[0076] (in (VQ is the set of indexes for the masked image patches)

[0077] Text-Image Alignment (TIA): Determines whether a text-image pair matches. Its loss function is... Cross-entropy loss for binary classification:

[0078]

[0079] (where H) CLSFor the output of the CLS marker, y=1 indicates a match.

[0080] This invention improves upon this basic model by introducing a "Domain Adaptive Visual Adapter" (DSVA) to enhance E in equation (5). visual This part makes it more suitable for the visual characteristics of professional fields.

[0081] The pre-training method for the domain-adaptive vision adapter module preferably includes the following steps:

[0082] (1) Constructing a domain-specific feature dataset: First, construct a visual feature dataset specific to the field of hydrogeology. For example, harvest and collect approximately 10,000 image patches from existing archival materials. These patches contain specific visual elements that frequently appear in this field, such as:

[0083] Lithological symbols: such as specific filler symbols for sandstone, mudstone, and limestone.

[0084] Handwritten text: Handwritten annotations and numerical values ​​commonly found in archives.

[0085] Table lines: A unique (blurred or irregular) table line style found in archives.

[0086] (2) Define the self-supervised pre-training task: Self-supervised contrastive learning is used as the pre-training task for the DSVA module.

[0087] (3) Perform pre-training: Connect the domain-adaptive vision adapter module, for example, a lightweight 3-layer residual network, to a fixed (Frozen) general vision backbone network, such as ResNeXt-FPN.

[0088] Data augmentation: Perform two random data augmentations (such as random cropping, rotation, and color dithering) on ​​each image slice in the domain feature dataset to obtain a pair of images (v i ,v j ).

[0089] Feature extraction: The two images are processed through the visual backbone and DSVA modules respectively to obtain the enhanced feature vector (z). i ,z j ).

[0090] Loss function: Use a contrastive learning loss function (such as NT-Xent Loss). The goal of this loss function is to "bring closer" the two enhanced feature vectors (zi, zi) of the same original image (such as the "sandstone" example) in the feature space. i ,z jAt the same time, it "pushes away" the feature vectors of different original images (such as the "sandstone" legend and the "handwritten signature").

[0091] (4) Application: After pre-training, the domain-adaptive visual adapter module has the ability to extract and enhance the refined features of specific visual elements in the hydrogeological field. Subsequently, the weights of this pre-trained domain-adaptive visual adapter module are loaded and concatenated into the complete model of this invention for subsequent overall model training.

[0092] Step S3: The OCR recognition module calls the OCR engine to recognize text in one or more logical regions and outputs the raw recognition data;

[0093] The OCR recognition module calls an OCR engine, such as Tesseract, PaddleOCR, or a commercial engine, to perform refined text recognition on the logical region determined in step S2, for example, a bounding box that is recognized as a "table".

[0094] A multimodal fusion recognition strategy is adopted, and both text area and table area recognition are based on sub-modules constructed from the improved LayoutLMv3 model to achieve accurate extraction of information from all types of data.

[0095] The specific improvements and design are as follows:

[0096] The first is the text region recognition submodule, which is connected to the BiLSTM-CRF layer after the improved LayoutLMv3 model to form the "improved LayoutLMv3-BiLSTM-CRF" architecture. The improved LayoutLMv3 is responsible for extracting the three-modal features of "text semantics + two-dimensional position + visual texture". The BiLSTM layer captures the text sequence dependency relationship (such as the term sequence of "Ordovician limestone") through formula (11). The CRF layer optimizes the global optimal solution of the label sequence through formula (12). Through this architecture, the accuracy of hydrogeological term recognition is improved to 96.5%, which is 8.3 percentage points higher than the original LayoutLMv3 model.

[0097] The second is the table area recognition submodule, which connects the improved LayoutLMv3 and LGPMA table structure recognition models. The improved LayoutLMv3 first outputs the text and position features of the table area, and the LGPMA model performs row and column line detection (Hough transform) and merged cell localization (watershed algorithm based on distance transform) based on these features. The accuracy of cell recognition is calculated by evaluating the correctness of cell merging and splitting, and finally a correspondence of "row index - column index - content" is established, so that the accuracy of table parsing with merged cells reaches 93.8%, which is 7.6 percentage points higher than the traditional "LayoutLMv3+LGPMA" solution.

[0098]

[0099] in This is the hidden state of the forward LSTM. For the backward LSTM hidden state, x t Let h be the feature of the t-th token in the input sequence. t This is a hidden state for bidirectional fusion.

[0100]

[0101] Where s(xi,yi,θ) is the score of the label sequence yi corresponding to the input sequence xi, Y(xi) is the set of all possible label sequences, and θ is the model parameter. Where s(xi,yi,θ) is the input sequence x.

[0102] Step S4: Data standardization module, based on domain knowledge base, performs data standardization processing on the original identification data;

[0103] The data standardization module relies on a specially constructed "hydrological professional knowledge base." This knowledge base stores professional terms, standard units, and the conventional numerical ranges of various parameters; for example, "aquifer thickness" is typically in the range of -500m to 1000m. The data standardization module uses this knowledge base to standardize the raw identification results from step three.

[0104] By constructing a professional knowledge base and designing targeted information extraction logic, professional processing and error correction optimization of OCR recognition results are achieved. The professional knowledge base is constructed in two main parts: a terminology database and a table template database. The terminology database is based on the "Specifications for Hydrogeological Exploration in Coal Mines" (GB / T 40130-2021), containing over 1200 core hydrogeological terms and establishing a "terminology-spelling-English comparison" relationship, covering key areas such as aquifer type, geological age, and hydrological parameters. The table template database provides standardized templates for 15 commonly used tables, including borehole record tables, water quality analysis tables, and hydrological observation logs, storing information such as table row and column headings, data types (e.g., numerical, text), and unit formats (e.g., "m³ / h", "mD"), providing a basis for table data verification. In terms of information extraction logic design, a dual mechanism of "terminology matching and error correction + table data verification" is adopted: the terminology matching stage compares the OCR recognition results with the terminology database through a fuzzy matching algorithm, and automatically corrects common errors such as "Ordovician-Ordovician Creek" and "water-conducting fracture zone-water-conducting fracture zone"; the table data verification stage combines the preset rules in the table template library to make reasonable judgments on the extracted values ​​(such as "water level elevation" is usually in the range of -500m to 1000m, and the unit of "permeability coefficient" must be "m / d"), and marks outliers that exceed the reasonable range or have incorrect units, so as to ensure the professionalism and accuracy of the data.

[0105] Step S5: The result verification and feedback module verifies the standardized data and optimizes the improved multimodal layout analysis model based on the verification results through closed-loop feedback.

[0106] A two-layer verification mechanism of "automatic verification + manual proofreading" is constructed to ensure the accuracy of the recognition results and to continuously optimize the LayoutLMv3 model through manual feedback. In the automatic verification stage, the system comprehensively compares the multimodal OCR recognition results with the hydrogeological professional knowledge base, and performs verification from three dimensions: terminology accuracy (matching the terminology base), numerical rationality (verifying the range and unit of numerical values), and table structure integrity (checking the number of rows and columns and merging cells). The overall accuracy of automatic verification can be calculated by formula (13) to evaluate the accuracy of the system, automatically generate a problem list, and mark the problem areas such as terminology errors, numerical anomalies, and table structure mismatches, providing clear guidance for manual proofreading.

[0107] In the manual proofreading stage, the system provides a visual interactive interface, using red highlighting to mark the automatically verified problem areas (highlighting transparency set to 50%), allowing hydrogeological staff to directly modify the erroneous content on the interface and save the correction results. At the same time, the manually corrected data is automatically fed back to the model training library as samples for semi-supervised learning to improve the parameter iteration optimization of the LayoutLMv3 model (iterates once for every 1000 correction data accumulated, with the learning rate decaying to 1e-6), gradually improving the system's adaptive recognition capability for complex scenarios (such as handwritten annotations and blurred handwriting), forming a closed-loop iterative mechanism of "recognition-verification-optimization". Its automatic verification comprehensive accuracy can be calculated by formula (13):

[0108] A ccauto-check =ω1Acc term +ω2Acc value +ω3Acc table (13)

[0109] Among them, Acc term For terminology accuracy, Acc value For numerical reasonableness and accuracy, Acc table The table structure accuracy is represented by ω1, ω2, and ω3, which are the weighting coefficients for each item (for example, based on experience, they can be set to ω1 = 0.4, ω2 = 0.4, and ω3 = 0.2).

[0110] The system automatically generates a list of issues based on this verification result; the manual verification process provides a visual interface, for example, highlighting abnormal data in red with 50% transparency, and collecting correction results. Every 1000 accumulated correction results are used to iteratively optimize the parameters of the model in step two through semi-supervised learning, thereby continuously improving model performance.

[0111] The specific improvements to the "improved LayoutLMv3 model" mentioned in step S2 are as follows:

[0112] Improved LayoutLMv3 model: Introducing a "Domain-Adaptive Visual Adapter"

[0113] The original LayoutLMv3 model uses a shared Transformer encoder to fuse features from three modalities: text, layout (position), and visual (image).

[0114] In this embodiment, the "improvement" is reflected in the addition of a "Domain-Specific Visual Adapter" (DSVA) module.

[0115] The domain-adaptive vision adapter module is a lightweight convolutional neural network, for example, consisting of 3 residual blocks (ResNetBlocks), which is cascaded between the visual backbone network of the original model, such as the fusion layer of ResNeXt-FPN and the Transformer encoder.

[0116] Its workflow is as follows:

[0117] Raw visual feature extraction: The raw document image is input into a visual backbone network, such as ResNeXt-FPN, to generate a preliminary visual feature map.

[0118] Domain feature enhancement: The preliminary visual feature map is not directly fed into the Transformer, but is first fed into the domain adaptive visual adapter module of this invention.

[0119] The domain-adaptive visual adapter module is pre-trained for specific visual elements in the field of hydrogeology, such as specific lithological symbols in borehole columnar plots, textures of handwritten annotations, and unique line styles in archival forms.

[0120] The domain-adaptive visual adapter module enhances and highlights the features of specific elements through convolution operations, while suppressing general (irrelevant) visual noise.

[0121] Output Enhanced Features: The Domain Adaptive Visual Adapter module outputs a "domain-enhanced" visual feature map.

[0122] Subsequent fusion: The "enhanced feature map" is flattened and divided into feature patches for subsequent fusion.

[0123] This improvement allows the model to quickly and efficiently adapt to the specialized visual features of the hydrogeological field without having to train a massive visual backbone from scratch. Instead, it can do so through a lightweight domain-adaptive visual adapter module, significantly improving the recognition accuracy of blurred scans, handwritten text, and specific legends.

[0124] In this invention, the fusion of the three features is completed at the input stage of the Transformer encoder, specifically through "EmbeddingSummation".

[0125] Text embedding: The text content is converted into a one-dimensional text feature vector through initial OCR or tokenization.

[0126] LayoutEmbedding: The coordinates (x0, y0, x1, y1) of the two-dimensional bounding box of each text unit (Token) are converted into a one-dimensional layout feature vector.

[0127] Enhanced Visual Embedding: The "enhanced visual feature map" processed by the domain-adaptive visual adapter module described above extracts the visual features corresponding to each text unit position and converts them into a one-dimensional visual feature vector.

[0128] Fusion: Before being input into the first layer of the Transformer, the three sets of feature vectors are added element-wise to form a unified multimodal fusion embedding vector.

[0129] Final_Input_Embedding=Text_Embedding+Layout_Embedding+Enhanced_Visual_Embedding

[0130] This "Final_Input_Embedding" is then fed into a multi-layered Transformer encoder for deep feature fusion and contextual understanding.

[0131] To achieve data standardization in step S4 and result verification in step S5, those skilled in the art can implement this "Hydrogeological Professional Knowledge Base" based on the following data structure example.

[0132] The "Hydrogeological Expertise Base" can preferably be a document database (such as MongoDB) or a JSON file, and its data structure is shown in the following example:

[0133]

[0134]

[0135]

[0136] In step S4, the data standardization module:

[0137] Scenario: In step S3, the OCR module identifies a table item as `{"Key":"Penetration coefficient","Value":"0.05"}`.

[0138] Retrieve: Using "penetration coefficient" as the keyword, search the knowledge base `Rules` in the "Data Standardization Module".

[0139] implement:

[0140] The retrieved rule is: `{"unit":"m / d","action":"standardize_unit"}`.

[0141] When performing the `standardize_unit` action: The system detected that the identified "Value" ("0.05") was missing a unit.

[0142] Output: The system automatically completes the standard unit "m / d" and outputs structured data: `{"Key":"permeability coefficient","Value":"0.05m / d"}`.

[0143] In the result verification module of step S5:

[0144] Scenario: In step S3, the OCR module identifies `{"Key":"Aquifer thickness","Value":"2000m"}`.

[0145] Retrieve: The "Result Verification Module" searches the knowledge base `Rules` using "aquifer thickness" as the keyword.

[0146] implement:

[0147] Retrieval rules:

[0148] `{"range":[-500,1000],"validation_logic":"check_range"}`.

[0149] Executing the `check_range` logic: `2000`(m) is not within the range of `[-500,1000]`(m).

[0150] Output: Verification failed. The system triggers the `flag_if_out_of_range` action, marking the data ("2000m") as an "outlier" and highlighting it in red with 50% transparency in the manual verification interface (as in step five), awaiting manual confirmation.

[0151] The following provides a complete embodiment demonstrating the complete process of processing an unstructured file using the method of the present invention.

[0152] Input: The user (step S1) uploads a scanned JPG image of the "ZK-101 borehole columnar section". The image includes a title, a descriptive text, and a table containing lithological data.

[0153] Forms (mixed handwriting and printing):

[0154] Depth (m) Lithology Aquifer thickness (m) 0-20 Clay 20 20-50 Sandstone 30 50-80 Mudstone 900 80-100 Limestone 20

[0155] Method execution flow:

[0156] Step S1: Receive input.

[0157] The system receives the image file `ZK-101.jpg`.

[0158] Step S2: Multimodal layout analysis.

[0159] The "Improved LayoutLMv3 model" was invoked.

[0160] The domain-adaptive visual adapter module, which has been pre-trained on hydrogeological legends, recognizes that the handwritten "sandstone" and the printed "clay" in the table have similar domain meanings, while also enhancing the table's FaintLines feature.

[0161] The model integrates text ("ZK-101"), layout (table BBox), and DSVA-enhanced visual features.

[0162] Output: The image was successfully segmented into three logical regions: [Title: ZK-101 borehole bar chart], [Text_Block: Static water level: 25.5 meters], [Table_Block: (Table region)].

[0163] Step S3: OCR recognition.

[0164] The OCR engine processes the three regions separately.

[0165] Output (raw recognition result):

[0166] {

[0167] Title: "ZK-101 Drilling Columnar Section",

[0168] "Text":"Still water level: 25.5 meters",

[0169] "Table":[

[0170] {"Depth (m)":"0-20","Lithology":"Clay","Aquifer Thickness (m)":"20"},

[0171] {"Depth (m)":"20-50","Lithology":"Sandstone","Aquifer Thickness (m)":"30"},

[0172] {"Depth (m)":"50-80","Lithology":"Mudstone","Aquifer Thickness (m)":"900"},

[0173] {"Depth (m)":"80-100","Lithology":"Limestone","Aquifer Thickness (m)":"20"} ]

[0175] }

[0176] Step S4: Data standardization module.

[0177] The module starts and queries the knowledge base.

[0178] `"Static water level: 25.5 meters": The system detected "meters" and standardized it to the standard unit "m" in the knowledge base. Output: `{"Key":"Static water level","Value":"25.5m"}`.

[0179] `"Aquifer thickness (m)":"20"`: The header "(m)" is detected, associated with the value "20", and the output is standardized as `{"Key":"Aquifer thickness","Value":"20m"}`. (The same applies to other rows.)

[0180] `"Lithology":"Clay": Query the knowledge base `Rules.Lithology.allowed_values` to confirm that "clay" is an allowed value.

[0181] Step S5: Result verification and feedback module.

[0182] The module starts and automatically verifies the standardized data.

[0183] Verify lines 1, 2, and 4:

[0184] `"Lithotype":"Clay"`->In `allowed_values`->Pass.

[0185] `"Aquifer thickness":"20m"`->Within `range:[-500,1000]`->Pass.

[0186] The same logic applies to ("sandstone","30m") and ("limestone","20m") -> Pass.

[0187] Verification line 3:

[0188] `"Lithology":"Mudstone"`->In `allowed_values`->Pass.

[0189] `"Aquifer thickness":"900m"`->Within `range:[-500,1000]`->Pass.

[0190] (Change the scenario to show the anomaly) Suppose that in the original table, "900" in row 3 is incorrectly identified as "9000" by OCR.

[0191] Verify line 3 (abnormal):

[0192] `"Aquifer thickness":"9000m"`->Not within `range:[-500,1000]`->Fail (according to the rules of Example 2).

[0193] action:

[0194] The system automatically generates a list of issues, marking "9000m" in line 3 as "numerical anomaly".

[0195] In the manual proofreading interface, this cell is highlighted in red (e.g., with 50% transparency).

[0196] Hydrogeologists reviewed the original image and corrected "9000" to the correct "900".

[0197] The system saves this correction record.

[0198] `(Image_Patch_of_"9000",Correct_Label_"900")`.

[0199] This record is fed back to the model training library for semi-supervised learning and iterative optimization of the model, with one iteration of optimization every 1000 records.

[0200] Server environment configuration:

[0201] To meet the computational requirements of improving the LayoutLMv3 model, the deployment environment adopts a "CPU+GPU collaborative computing" architecture, with the following specific configuration: The server hardware consists of an Intel Xeon E5-2680 CPU (16 cores, 32 threads, 2.8GHz) and an NVIDIA Tesla V100 GPU (32GB VRAM, 15.7 TFLOPS computing power) to ensure high efficiency in model inference and data computation; the operating system is Ubuntu 20.04LTS, offering good compatibility and stability; the development framework uses PyTorch 1.12, coupled with OpenCV 4.5.5 (image preprocessing), Hugging Face Transformers (improving LayoutLMv3 model loading), and EasyOCR 1.6.2 (basic text recognition) libraries; the database uses a MySQL 8.0 master-slave architecture, with the master database responsible for data writing and the slave database responsible for data reading, ensuring data security and traceability (data read / write latency <1s).

[0202] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0203] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for rapid document entry and recognition based on multimodal adaptive OCR, characterized in that, Includes the following steps: Step S1: Receive the image file of the unstructured archive; Step S2: An improved multimodal layout analysis model is used to perform layout analysis on the image file. The improved multimodal layout analysis model integrates text semantic features, two-dimensional position features, and domain-enhanced visual features to segment the image file into one or more logical regions. Step S3: Call the OCR engine to perform text recognition on the one or more logical regions and output the original recognition data; Step S4: Based on the domain knowledge base, perform data standardization processing on the original identification data; Step S5: Verify the results of the standardized data, and optimize the improved multimodal layout analysis model based on the verification results using closed-loop feedback.

2. The method for rapid document entry and recognition based on multimodal adaptive OCR according to claim 1, characterized in that: The improved multimodal layout analysis model is constructed by connecting a Domain Adaptive Visual Adapter (DSVA) module in series on a basic multimodal model. The Domain Adaptive Visual Adapter module is configured to perform domain feature enhancement on the preliminary visual features of the image file before the basic multimodal model fuses features, so as to output the domain-enhanced visual features.

3. The method for rapid document entry and recognition based on multimodal adaptive OCR according to claim 2, characterized in that: The domain-adaptive visual adapter module is a lightweight convolutional neural network pre-trained for specific visual elements in a specific professional field.

4. The method for rapid document entry and recognition based on multimodal adaptive OCR according to claim 1, characterized in that: The domain knowledge base stores the terminology, standardization rules, and data verification rules used for the data standardization process and the result verification.

5. The method for rapid document entry and recognition based on multimodal adaptive OCR according to claim 4, characterized in that: The standardization rules and data verification rules include: a list of allowed values ​​for technical terms; and the type, unit, and / or reasonable range of numerical data.

6. The method for rapid document entry and recognition based on multimodal adaptive OCR according to claim 1, characterized in that: Step S4 includes: performing terminology correction and / or unit completion on the original identification data based on the domain knowledge base.

7. The method for rapid document entry and recognition based on multimodal adaptive OCR according to claim 1, characterized in that: Step S5 includes: automatically screening the standardized data based on the data verification rules in the domain knowledge base to mark outliers; highlighting the outliers in the visualization interface to receive manual correction; closed-loop feedback optimization includes: collecting the results of the manual correction and using the results as semi-supervised learning samples to perform parameter iterative optimization of the improved multimodal layout analysis model.

8. The method for rapid document entry and recognition based on multimodal adaptive OCR according to claim 2, characterized in that: The textual semantic features, two-dimensional positional features, and domain-enhanced visual features are fused by summing through an embedding layer.

9. A rapid document entry and recognition system based on multimodal adaptive OCR, characterized in that, Includes modules configured to perform the method of any one of claims 1 to 8: The input receiving module is configured to receive image files from unstructured archives; The multimodal layout analysis module is configured to perform layout analysis on the image file using an improved multimodal layout analysis model; The OCR recognition module is configured to invoke the OCR engine for the one or more logical regions; The data standardization module is configured to perform data standardization processing on the original identification data based on a domain knowledge base. The result verification and feedback module is configured to verify the standardized data and perform closed-loop feedback optimization on the improved multimodal layout analysis model based on the verification results.

10. A document entry and recognition system based on multimodal adaptive OCR according to claim 9, characterized in that, The multimodal layout analysis module includes a basic multimodal model and a domain-adaptive visual adapter module connected between the visual backbone network and the fusion layer of the basic multimodal model. The domain-adaptive visual adapter module is configured to perform domain feature enhancement on the preliminary visual features extracted by the basic multimodal model.

Images