A multi-modal document parsing method, electronic device and storage medium

Abstract

The application provides a multi-modal document parsing method, an electronic device and a storage medium, and the method comprises the following steps: preprocessing an original document, identifying key elements (at least including formulas, tables, texts and images) by using a target detection model to obtain an element positioning label table; obtaining a mark based on the label table and a coarse recognition task type, combining original document metadata to identify a field, and generating field adaptation parameters; defining an element range according to the label table, combining the task mark to detect obstacles in layers and repairing obstacles in layers, and obtaining an element document without obstacles; cooperatively coding the element document without obstacles, splitting sub-tasks based on the task mark and a coding result for parallel processing, calling the adaptation parameters to adjust the accuracy; checking and correcting a processing result, and integrating the processing result into a final structured document report according to the adaptation parameters. The application can improve the efficiency and accuracy of multi-modal document parsing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and document information processing technology, in particular to a multi-modal document parsing method, an electronic device and a storage medium. BACKGROUND

[0002] Under the background of accelerating digital transformation, multi-modal documents (containing formulas, tables, and mixed text and graphics elements) such as academic papers, corporate financial reports, medical records, and engineering drawings have become the core force of information carriers, and the number of documents has increased exponentially. Document parsing technology aims to automatically and accurately extract structured and unstructured information from documents, providing key support for knowledge management, intelligent decision-making, and cross-domain data fusion.

[0003] Currently, document parsing technology faces many challenges. In the field of formula processing, mainstream methods (such as CNN+RNN-based architecture or Mathpix tools) rely heavily on fixed symbol libraries, and their recognition performance significantly decreases when dealing with complex handwritten formulas, custom symbols, or special formulas in professional fields. In the field of table processing, rule-based methods (such as PDFMiner) struggle to handle complex structures, while traditional computer vision methods (such as edge detection + Hough transform) often misjudge the structure in scenarios involving merged cells, nested tables, or blurred table lines. In the field of multi-modal information extraction, existing technologies often use simple feature concatenation methods (such as ResNet features and BERT vectors), which fail to achieve deep semantic alignment. Pre-trained language models (such as DocBERT) also only recognize non-text elements (such as graphs and formulas) in isolation, without deep mining of the logical relationships between elements.

[0004] In recent years, although some research has attempted to introduce domain adaptation mechanisms or end-to-end architectures (such as LayoutLM), there are still some core limitations: first, the model parameters cannot be dynamically adjusted to adapt to different industries' special formats, leading to a sharp decline in recognition accuracy; second, the model lacks intelligent completion mechanisms for complex scenarios such as occlusion, blur, and missing elements; third, the modules are not well coordinated, and the features are lost during transmission, resulting in low completeness and accuracy of multi-modal information extraction. These problems severely restrict the large-scale application of the technology in research, finance, and healthcare. SUMMARY

[0005] To address the above technical problems, the technical solution adopted by the present application is as follows:

[0006] According to the first aspect of the present application, a multi-modal document parsing method is provided, which comprises the following steps:

[0007] S100, performing a preprocessing operation on an original document to generate a preprocessed document, identifying all key elements in the preprocessed document by using a target detection model to obtain an element positioning annotation table; the key elements at least include formulas, tables, texts and images, and the element positioning annotation table contains an element ID, coordinate information and a preliminary type corresponding to each key element.

[0008] S200, performing task type coarse recognition based on the element positioning annotation table, determining a task type identifier of the original document according to the coarse recognition result; identifying the field to which the original document belongs based on the metadata of the original document, and generating a field adaptation parameter in combination with the task type identifier and the field.

[0009] S300, determining the area range of each key element according to the element positioning annotation table, performing hierarchical obstacle detection in combination with the task type identifier to obtain the obstacle type of all key elements; calling a repair logic corresponding to the obstacle type for targeted processing to generate an obstacle-free element document that retains the coordinate association of each element with the original document.

[0010] S400, performing collaborative coding on all key elements in the obstacle-free element document to generate a collaborative coding result; generating a task token based on the task type identifier and the collaborative coding result, splitting specific subtasks according to the task token and performing parallel processing, calling the field adaptation parameter to adjust the processing precision during processing, and generating a multi-task parallel processing result set.

[0011] S500, performing multi-dimensional verification and correction on the multi-task parallel processing result set, integrating the corrected result according to the format requirement in the field adaptation parameter to form a final structured document report. According to the second aspect of the present application, an electronic device is provided, which includes a processor and a memory; the processor is used to execute the steps of the method of the first aspect of the present application by calling the program or instruction stored in the memory.

[0012] According to the second aspect of the present application, a computer readable storage medium is provided, which stores a program or instruction, and the program or instruction makes the computer execute the steps of the method of the first aspect of the present application.

[0013] The multi-modal document parsing method provided by the present application has at least the following beneficial effects:

[0014] (1) Hierarchical obstacle detection and targeted repair logic, avoiding excessive processing of non-key elements, greatly improving obstacle processing efficiency, while ensuring the repair accuracy of key elements (such as formulas and tables) and reducing parsing errors;

[0015] (2) The field adaptation parameter combines the task type to dynamically adjust the processing precision, so that the method can adapt to different field (such as mathematics, finance, medical) documents, and solve the problem of insufficient generality of traditional methods;

[0016] (3) The cooperative coding is combined with the sub-task parallel processing to strengthen the multi-element correlation analysis ability, speed up the processing speed, and adapt to large-scale multi-modal document parsing scene;

[0017] (4) The elements are associated with the original document coordinates throughout the process, the repair and analysis results are traceable, the final structured report meets the downstream needs of editing, searching and the like, and the utilization value of the document is improved.

[0018] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A flowchart of a multi-modal document parsing method provided by the embodiments of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0023] It is noted that some example embodiments are described as processes or methods depicted as flowcharts. Although the processes are described in a particular, sequential order, many of the steps can be performed in parallel, concomitantly, or concurrently. Further, the order of the steps can be re-arranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the figure. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0024] To effectively solve the problems of low accuracy and poor efficiency in processing complex documents in the prior art, and significantly improve the intelligent level and application value of document processing, an embodiment of the present application provides a multi-modal document parsing method, as shown in Figure 1 The method comprises the following steps:

[0025] S100, performing a preprocessing operation on an original document to generate a preprocessed document, using a target detection model to identify all key elements in the preprocessed document, and obtaining an element positioning annotation table; the key elements at least include formulas, tables, texts, and images, and the element positioning annotation table contains element IDs, coordinate information, and preliminary types corresponding to each key element.

[0026] In the embodiment of the present application, the original document includes but is not limited to PDF documents, Word documents, scanned copies, and image documents. The preprocessing operation on the original document includes: first, converting the original document into a standard parseable format, the standard parseable format being selected from at least one of a high-definition image sequence or a structured document stream; for scanned copies and image documents, further performing image optimization operations of denoising, sharpening, and tilt correction to improve the clarity of the document content; calling the target detection model to traverse the preprocessed document area, identifying and marking all key elements in the document; recording the element ID, preliminary type, and coordinate information of each key element to form an element positioning annotation table, the coordinate information being in the form of (x1, y1, x2, y2) representing the boundary range of the element in the document, and finally outputting the preprocessed document and the element positioning annotation table.

[0027] Further, the preprocessing operation further includes resolution unification and size standardization processing on the converted high-definition image sequence or structured document stream to ensure the consistency of the subsequent target detection model input. For structured document streams (such as HTML or XML format), label cleaning and redundant element elimination are also performed to improve the accuracy of key element positioning.

[0028] In the image optimization operation, the denoising processing uses a non-local mean denoising algorithm or a wavelet threshold denoising method; the sharpening processing uses a Laplacian operator or a non-sharpening mask; the tilt correction is based on Hough transform or projection profile analysis for automatic detection and rotation correction.

[0029] Before calling the target detection model, the pre-processed document is also subjected to blocking processing, which is particularly suitable for large-size or complex layout documents. An overlapping sliding window strategy is adopted to ensure the integrity of cross-block elements.

[0030] The target detection model can adopt YOLOv5, YOLOv7 or higher version of YOLO series model, or be replaced by FasterR-CNN, DETR and other structures, and be pre-trained and fine-tuned on specific document data sets to improve the recognition accuracy of complex elements such as formulas and tables.

[0031] The element positioning label table also contains the confidence score of each key element and the page number (for multi-page documents), and can be output in JSON or XML format for subsequent processing system integration and parsing.

[0032] In addition, after forming the element positioning label table, the preliminary classification results are subjected to post-processing verification, and the element type is subjected to secondary verification through a rule engine or a lightweight classifier, especially for easily confused types between formulas and tables, text and images, to improve the robustness of type judgment.

[0033] The final output of the pre-processed document and the element positioning label table will be used for downstream tasks such as document understanding, content reorganization, intelligent retrieval or accessibility support.

[0034] S200, based on the element positioning label table, performing task type coarse recognition, determining the task type identifier of the original document according to the coarse recognition result; identifying the field to which the original document belongs based on the metadata of the original document, and generating field adaptation parameters in combination with the task type identifier and the field.

[0035] Further, the task type coarse recognition based on the element positioning label table, and the determination of the task type identifier of the original document according to the coarse recognition result, specifically includes:

[0036] Statistically, the proportion of the number of formula elements in the element positioning label table to the total number of all identifiable elements is greater than or equal to the first preset proportion, and the formula processing task type identifier is activated. The first preset proportion can be set to 40%.

[0037] The table elements in the element positioning label table are analyzed, and if the table elements satisfy any of the following conditions: the proportion of the number of merged cells to the total number of cells of the corresponding table is greater than or equal to the second preset proportion, the number of rows of the table is greater than the preset number of rows, and the table contains multi-modal content such as formulas or images, then the table processing task type identifier is activated. The second preset proportion can be set to 20%, and the preset number of rows can be set to 50.

[0038] If the proportion of the formula element quantity in the total number of all identifiable elements is less than the first preset proportion, that is, the proportion of the formula element quantity does not reach the first preset proportion, and all table elements do not satisfy any of the above conditions, the text processing task type identifier is activated.

[0039] In the embodiment of the application, the metadata of the original document at least includes a title, keywords, an author unit, a document abstract and file attribute information.

[0040] Further, the field to which the original document belongs is identified based on the metadata of the original document, and specifically includes:

[0041] S201, performing normalization processing on the metadata of the original document, including performing word segmentation and stop word removal on text metadata (such as a title, an abstract and keywords), and uniformly standardizing mapping of format tags.

[0042] S202, calling a field general model pre-trained based on multi-field corpus, performing feature extraction on the normalized metadata, and outputting a corresponding field feature vector.

[0043] In the embodiment of the application, the field general model is constructed based on an improved Transformer encoder. The improved encoder embeds a learnable field parameter matrix as an independent "adaptation layer" after the original feature encoding step of the traditional Transformer encoder and before the feedforward neural network processing.

[0044] The dimension of the learnable field parameter matrix can be represented as [field feature dimension x model hidden layer dimension]. Among them, the number of rows corresponds to the "field feature dimension", which represents the number of specific field core feature types to be covered, such as the "∀" and "∃" symbols unique to the mathematical field, the "ρ" and "Ω" symbols in the physical field, and the table structure features (such as the "same period / compared period" column layout in the financial statement, the "symptom-index" row association relationship in the medical table, etc.); the number of columns corresponds to the "model hidden layer dimension", that is, the feature vector dimension output by the Transformer encoder. This design fuses the weights of the field-specific features and the general semantic features extracted by the model through matrix operations (such as matrix multiplication), realizing the mapping from "field features" to "model semantic space".

[0045] Since the matrix parameter is learnable, the application adopts a meta-learning strategy to train it, and the process includes three stages of pre-training, task design and parameter optimization.

[0046] Pre-training stage: By collecting annotated data in multiple fields such as mathematics, physics, finance, medicine, and chemical engineering, the Transformer model with embedded field parameter matrix is preliminarily trained to enable the model to grasp the common rules of different field characteristics, such as formula symbol semantic representation weight distribution and table structure general feature extraction logic, thereby building the foundation for cross-field analysis.

[0047] Task design stage: Each field is considered as an independent task, simulating a small sample learning scenario, and providing a small amount of annotated samples for each task (such as 500-1000 formula samples containing specific symbols for the chemical engineering field), requiring the model to quickly adjust parameters to adapt to the corresponding field.

[0048] Parameter optimization stage: The initial parameters of the model are optimized to enable it to adapt to new fields with only a small amount of samples and gradient descent steps. During training, the model makes predictions based on the current parameters, calculates the loss based on the difference between the predicted results and the annotated samples, and updates the parameters. The initial parameters are further optimized by integrating the update experience of multiple tasks to improve their generalization ability.

[0049] The adapter serves as a general module, achieving multi-field adaptation by dynamically adjusting the field parameter matrix. When processing new field documents, based on the meta-learning strategy, 500-1000 annotated samples are used to update the parameters: the model uses the current field parameter matrix to process the samples, calculates the loss between the prediction and the annotation, and adjusts the matrix parameters based on the gradient to enhance the response strength to the target field characteristics, thereby achieving precise alignment of global semantic representation and field characteristics.

[0050] The field-universal model serves as a teacher model, taking massive, multi-modal, and cross-field universal documents as input, and outputting structured multi-modal analysis and association results in JSON, XML, etc. During the training stage, the teacher model is pre-trained on massive cross-field corpora using the Transformer encoder to extract the general semantic features of text, formula, table, and other multi-modal data. This process uses a self-supervised learning mechanism to strengthen the learning of general patterns through masked language modeling and formula structure prediction tasks.

[0051] By jointly optimizing multi-objective losses such as formula analysis loss (difference between predicted symbols and true annotations), table structure loss (difference between predicted row-column relationships and true relationships), and association prediction loss (difference between predicted text and other element association types and true associations), and using gradient descent algorithms such as Adam, the model is iteratively trained on the general training set until it converges, enabling it to stably output analysis results consistent with cross-field logic.

[0052] The specific adjustment of the domain parameter matrix is implemented by a student model. The student model adopts a knowledge distillation strategy and quickly masters domain-independent knowledge such as formula syntax rules and table row and column logic by comparing the output of the teacher model. The student model also receives multi-modal inputs such as text, formulas, tables, and images, but needs to be preprocessed for specific domain documents (such as engineering drawings containing special symbols and handwritten medical records) such as symbol standardization and handwriting enhancement.

[0053] In the feature extraction stage, the student model parameter is initialized as the teacher model weight, and the text, formula, table, and image encoders thereof are reused. At the same time, the domain feature adapter updates the parameters of the specific domain features. For example, in formula analysis, the student model learns the analysis mode of the teacher model for LaTeX syntax structure; in table processing, the recognition logic of the cell nesting relationship is inherited. In addition, by introducing a domain-specific knowledge graph and an industry term mapping table, the student model can dynamically adjust the parameter weight and construct a professional knowledge network for a specific domain (such as financial reports and academic papers).

[0054] S203, the domain feature vector is matched with the reference features of each domain in the preset domain feature library, the cosine similarity between the domain feature vector and each domain reference feature vector in the library is calculated, and a group of similarity scores is obtained.

[0055] S204, if the maximum similarity score is greater than or equal to the first preset threshold, and the difference between the maximum similarity score and the second largest similarity score is greater than or equal to the preset difference threshold, the original document is directly determined to belong to the domain corresponding to the maximum similarity score.

[0056] In the embodiment of the application, the first preset threshold can be set to 0.85, and the second preset threshold can be set to 0.75. The preset difference can be set to 0.05.

[0057] S205, if there are multiple similarity scores greater than or equal to the second preset threshold, and the difference between these similarity scores is less than the preset difference threshold, it is determined that the multi-domain similarity is close, and the author's unit information in the metadata is combined for calibration, and finally a unique domain identifier is output.

[0058] In the embodiment of the application, the second preset threshold can be set to 0.75. When the multi-domain similarity is close, the author's unit information in the metadata is combined for calibration. By querying the pre-built “unit-domain” mapping knowledge base, the association confidence of each candidate domain with the author's unit is calculated. The domain with the highest association confidence is determined as the final domain identifier. If the author's unit information is missing or cannot be matched, the domain with the highest similarity score is selected as the default output.

[0059] In the embodiment of the application, the calibration process specifically includes:

[0060] (1) Construct a "unit-field" prior mapping library: a knowledge base is pre-constructed to store a large number of mapping relationships between well-known institution names and their main research or business fields. The mapping library is usually stored in the form of key-value pairs and can include weights. For example, "A University Computer Science and Technology Department": {"field": "Computer Science", "weight": 0.95}.

[0061] (2) Analyze and match the author's unit: extract the author's unit field from the metadata, and through fuzzy string matching (such as using edit distance or semantic similarity matching based on pre-trained language models), query the most likely associated field and unit association weight of the unit in the "unit-field" mapping library.

[0062] (3) Perform weighted calibration decision: adopt a weighted scoring mechanism to calculate the final score of each candidate field, and select the field with the highest score as the output identifier. The final score = original similarity x a + unit association weight x b, where a and b are harmonic coefficients used to balance the weight of the original model confidence and the unit prior information, and a + b = 1, for example, both are set to 0.5. Assuming that a document is identified by the field general model, the "finance" field similarity is 0.85, and the "industry" field similarity is 0.83 (the difference 0.02 is less than the threshold 0.05, triggering calibration). The author's unit in the metadata is "XX Securities Research Institute". Through the "unit-field" mapping library query, the unit is strongly associated with the "finance" field (weight 0.95). The final score of "finance" is 0.85 x 0.5 + 0.95 x 0.5 = 0.9; The final score of "industry" is 0.83 x 0.5 + 0.0 x 0.5 = 0.415 (assuming that the unit has no association with "industry", the weight is 0). Obviously, "finance" wins.

[0063] (4) Default processing when there is no match: if the author's unit is not found in the mapping library or the unit information is missing, the field with the highest original similarity is selected as the output by default, and the log is recorded.

[0064] Further, the combination of the task type identifier and the field generates a field adaptation parameter, specifically including:

[0065] S210, according to the identified field, calling the initial parameter template of the corresponding field from the pre-constructed field parameter template library.

[0066] In the embodiment of the application, the initial parameter template in the pre-constructed field parameter template library is determined by the following offline training and archiving process:

[0067] Step 1, Building a domain-specific training dataset: For each target domain (such as mathematics, finance, medicine, and industry), collect and clean large-scale, high-quality professional document data to form a domain-specific training set $D_{domain}$. Fine annotation is performed on this dataset, including but not limited to: LaTeX sequence of formulas, table structure relationship, entity and relationship triplets, and element association annotations.

[0068] Step 2, Initialize a domain-general teacher model: Use a large amount of cross-domain, multi-modal general document data to pre-train the improved Transformer encoder (i.e., a model embedded with a learnable domain parameter matrix). The goal of this stage is to enable the model to learn cross-domain general feature representation and basic analysis capabilities, resulting in a powerful, general teacher model. The parameters of this model include its general domain parameter matrix.

[0069] Step 3, Domain-specific fine-tuning (knowledge distillation):

[0070] For each specific domain, create a corresponding student model based on the pre-trained general teacher model.

[0071] Use the domain-specific training set prepared in Step 1 to fine-tune the student model with supervised training. The core goal of fine-tuning is to adapt the model to the unique features of the domain. During training, other parameters of the model can be adjusted slightly, but the focus is on learning and optimizing the learnable domain parameter matrix.

[0072] The loss function used in training is a comprehensive loss, including formula analysis loss and table structure loss, to ensure high accuracy in the domain.

[0073] Step 4, Extract and archive domain parameter templates:

[0074] After domain-specific fine-tuning is complete, extract the domain parameter matrix from the student model that has been optimized and contains the core feature knowledge of the domain.

[0075] This matrix serves as the initial parameter template for the domain.

[0076] Store this initial parameter template along with its corresponding domain identifier (such as "finance" or "mathematics") in the domain parameter template library.

[0077] Step 5, Maintenance and expansion of the template library:

[0078] The initial template library contains parameter templates for the first batch of target domains.

[0079] When a new domain needs to be introduced, repeat steps 1-4 to generate and archive the initial parameter template of the new domain, thereby extending the template library.

[0080] S211, according to the task type identification, the initial parameter template is adjusted to obtain the adjusted domain parameter matrix initial weight. The purpose of this directional adjustment process is to make the initial state of the model more suitable for the main task type to be processed, so as to improve the subsequent fine-tuning efficiency and final effect.

[0081] Specifically, if the formula processing task type identification is identified, the feature dimension weight related to the formula symbol in the domain parameter matrix is adjusted, and the knowledge inheritance weight is adjusted to focus on the formula syntax analysis parameters of the domain general model; if the table processing task type identification is identified, the feature dimension weight related to the table structure in the domain parameter matrix is adjusted, and the precision threshold is adjusted according to the core requirements of the table data.

[0082] The knowledge inheritance weight is a scalar coefficient or a group of coefficients, which is used to control the relative importance between the general knowledge learned by the student model from the domain general teacher model and the specific knowledge learned from the current specific task data in the domain adaptation process. Its core function is to balance generality and specificity, and prevent "catastrophic forgetting" or overfitting in the process of fine-tuning a small amount of samples. High weight means that the general parameters and knowledge of the teacher model are more trusted and retained; low weight means that the parameters are more inclined to be updated according to the current task data.

[0083] Further, if the task type identification is "formula processing", first, a formula symbol mask (MaskF) of the same dimension as the domain parameter matrix (the mask is defined in the pre-training stage, and in the templates of mathematics, physics, etc. Mark those feature channels corresponding to "∑", "∫", "∂" and other operators or special symbols) is weighted. The adjustment operation can be represented as: W adapted =W initial +λ f ×MaskF⊙W initial . Where, W initial is the initial parameter template, W adapted is the adjusted weight, λ f is a task reinforcement coefficient greater than 0 (for example, set to 0.1), and ⊙ represents element-wise multiplication. The operation aims to amplify the sensitivity of the model to formula symbol features.

[0084] Next, the weight λ kdThe proportion in the overall loss function is increased (for example, from 0.5 to 0.7). This means that during fine-tuning, the feature distribution output by the student model needs to maintain a higher similarity with the output distribution of the teacher model for the same formula, thereby forcing the student model to inherit the strong formula syntax analysis prior knowledge of the teacher model and avoiding learning bias due to the small number of samples.

[0085] Further, if the task type is identified as "table processing", first, in the initial parameter matrix, the feature dimension weight related to the table structure is identified and increased. A table structure mask (MaskT) is used for enhancement, which marks the feature channels corresponding to the row-column relationship, cell merging mode, table header features, and other structure information. The adjustment operation can be represented as: W adapted =W initial +λ t ×MaskT⊙W initial . Where λ t is a reinforcement coefficient special for table tasks. This operation makes the model pay more attention to the geometric structure and logical relationship of the table during initialization.

[0086] Next, according to the core requirements of table data (such as the high accuracy requirement of financial statements for numerical values), the key thresholds in subsequent processing are adjusted, including:

[0087] Increase the IOU threshold of cell positioning: for example, from 0.8 to 0.9, requiring a higher degree of overlap between the predicted cell bounding box and the true box to improve coordinate extraction accuracy.

[0088] Increase the confidence threshold of structure relationship prediction: for example, from 0.7 to 0.85, only when the model predicts the confidence of "parent-child cells", "table header-data cells" and other relationships exceeds this threshold, it is determined that the relationship exists, thereby reducing the misjudgment rate and ensuring the rigor of table structure analysis.

[0089] Further, if the task type is identified as "text processing", the feature dimension weight related to text semantics and syntax patterns can be focused on, and the knowledge inheritance weight can be maintained or slightly reduced, because text processing tasks rely more on specific domain terminology and context, encouraging the model to learn more from the current document data.

[0090] Through the above directional adjustment strategy, the initial weight and configuration parameters of the domain parameter matrix customized for the current document and its core task can be obtained, laying the optimal foundation for subsequent efficient and accurate fine-tuning based on a small number of samples.

[0091] S212, adjusting the number of samples required for meta-learning fine-tuning according to the common degree in the field: the number of samples required in the common field is set to a first preset value, and the number of samples required in the unpopular or scarce field is adjusted to a second preset value, and the exclusive knowledge graph corresponding to the field is supplemented into the field knowledge enhancement sub-parameter.

[0092] In the embodiment of the application, the common field refers to a field with a large amount of open and easily accessible labeled data, a field with relatively standardized document format and content, such as mathematics (academic papers with a large number of LaTeX formulas), general finance (standard financial reports of listed companies, news), and basic medical care (standardized medical examination reports, common disease medical records). Because of the rich data, the model is easy to learn, and the number of samples required for fine-tuning is small. The first preset value is usually set to a lower order of magnitude, for example, 500 labeled samples.

[0093] The unpopular field refers to a field with high data acquisition difficulty, high labeling cost, or strong professional barriers, but it still has stable output and a certain amount of inventory data, such as geology (mineral report, geological exploration map), precision instrument (device manual, fault diagnosis log), and archaeology (literature excerpt, unearthed cultural relic record table). Because the data features are sparse or unique, more samples are needed to fully learn the characteristics of the field. The second preset value is usually set to several times the first preset value, for example, 2000 labeled samples.

[0094] The scarce field refers to a field with extremely scarce data, high sensitivity, or emerging fields, and it is almost impossible to obtain sufficient labeled data, such as specific military technology (technical specifications, design schemes), cutting-edge biological experiments (experimental manuscripts, unpublished reagent formula tables), and small-scale ancient language literature.

[0095] For the scarce field, it is difficult to effectively adapt by simply increasing the number of samples (even if the second preset value). At this time, it will rely heavily on the prior knowledge extracted from the exclusive knowledge graph of the field, and the meta-learning fine-tuning process is more like a "knowledge-based initialization" or "zero-shot / few-shot" learning, and the number of samples required may be only 100-300, but its main role is to calibrate and activate the knowledge graph information.

[0096] Further, the exclusive knowledge graph corresponding to the field is supplemented into the field knowledge enhancement sub-parameter, which specifically includes the following steps:

[0097] (1) Pre-construction of the exclusive knowledge graph of the field: an offline structured knowledge base is constructed for each unpopular or scarce field. The graph takes triplets (entity-relation-entity) or (property-entity-value) as the core form, and stores the core concepts, terminologies, symbols and their mutual relationships in the field. For example, the chemical industry graph:

[0098] (Ethylene, production process, cracking)

[0099] (reaction vessel, having properties, resistant to high pressure)

[0100] (chemical formula - h2so4, is, sulfuric acid)

[0101] (catalyst - pt, for, reaction - ammoxidation)

[0102] (2) Knowledge extraction and vectorization: Use knowledge graph embedding techniques (such as TransE, RotatE, etc.) to map entities and relationships in the graph to a low-dimensional dense vector space, obtaining vector representations of each entity (i.e., knowledge embedding). These vectors capture the semantic and structural information in the graph.

[0103] (3) Build a knowledge query module: Design a knowledge query module within the domain adaptation framework. This module receives entities or symbols (such as an unusual chemical formula or instrument number) preliminarily extracted from the current processing document as the query key (QueryKey).

[0104] (4) Supplement the domain knowledge enhancer parameters, including:

[0105] Knowledge retrieval: The knowledge query module uses approximate nearest neighbor (ANN) search and other techniques to quickly retrieve a number of entities and their vector representations most relevant to the query key in the pre-generated knowledge embedding vector library.

[0106] Knowledge injection: Inject these retrieved knowledge vectors as additional context information into the model's feature representation. This can be achieved in the following ways:

[0107] Concatenation: Concatenate the knowledge vector with the current hidden layer feature vector of the model, and then adjust the dimension through a linear projection layer.

[0108] Attention weighted sum: Treat the knowledge vector set as a group of "values (Value)" and "keys (Key)", and the model's current features as "queries (Query)", and calculate the weighted sum through the attention mechanism to dynamically integrate the most relevant knowledge information.

[0109] Parameterization: The adjustable parameters of the entire knowledge query and fusion module (such as the weights of the projection layer, attention weights, etc.) are collectively referred to as domain knowledge enhancer parameters. During the meta-learning fine-tuning process, these parameters are iteratively optimized together with the domain parameter matrix, enabling the model to learn how to effectively query and utilize relevant information from the knowledge graph to assist decision-making.

[0110] Through the above mechanisms, domain-specific knowledge graphs are effectively supplemented to the model, greatly enhancing the understanding and processing capabilities of rare and scarce domain content, and making up for the lack of training data.

[0111] S213, integrate the adjusted domain parameter matrix initial weight, meta-learning fine-tuning sample requirement, precision threshold, format rule and knowledge fusion parameter into a complete domain adaptation parameter file.

[0112] The adjusted domain parameter matrix initial weight is a learnable parameter matrix after task-oriented reinforcement (step S202). It is a floating-point matrix with the shape of [domain feature dimension x model hidden layer dimension]. This matrix will be used as the initial state of the fine-tuning process, rather than starting from zero. The meta-learning fine-tuning sample requirement is an integer value that specifies the number of labeled samples required for fine-tuning in this domain. Its value is determined by the domain commonality (step S203), for example: common domain = 500, niche domain = 2000, rare domain = 100. The precision threshold is a dictionary or JSON object that stores the confidence threshold for different task sub-modules. The format rule is a dictionary or JSON object that defines the standardized format of the output results. This rule is determined according to the domain convention and task type. The knowledge fusion parameter includes the knowledge inheritance weight and the domain knowledge enhancement sub-parameter. The knowledge inheritance weight is a floating-point scalar (such as 0.7) that is used to weight the importance of the knowledge distillation loss term in the fine-tuning loss function, controlling the degree of preservation of the teacher model's general knowledge (step S202). The domain knowledge enhancement sub-parameter is a complex parameter that includes information derived from the domain-specific knowledge graph supplemented in step S203. It includes the knowledge graph embedding vector library path and the knowledge query parameter. The knowledge graph embedding vector library path points to a pre-generated entity / relation vectorization file (such as a.bin or.pt file) of the domain knowledge graph. The knowledge query parameter includes the relevant configuration of the knowledge retrieval and fusion module, such as the index type of the approximate nearest neighbor (ANN) search, the number of most relevant entities returned (top-k), and the initial weight of the attention fusion layer, etc.

[0113] The above components are collectively encapsulated in a top-level structured configuration object (usually in a readable format such as JSON or YAML), forming a complete, machine-readable domain adaptation parameter configuration file. This file is one of the final outputs of the dynamic domain adaptation framework and will be loaded and executed by the subsequent meta-learning fine-tuning engine and inference engine.

[0114] S300, determine the area range of each key element according to the element positioning annotation table, perform hierarchical obstacle detection combined with the task type identifier, and obtain the obstacle type of all key elements; call the corresponding repair logic for targeted processing, and generate an obstacle-free element document that preserves the association between each element and the original document coordinates.

[0115] In the embodiment of the present application, the execution of hierarchical obstacle detection in combination with the task type identification includes: performing fine obstacle detection on the key element area matched with the task type identification, and performing basic obstacle detection on the non-key element area.

[0116] Further, the execution of fine obstacle detection on the key element area matched with the task type identification and the execution of basic obstacle detection on the non-key element area specifically include:

[0117] S301, for the key element area matched with the task type identification, calling a real-time obstacle detection framework constructed based on a YOLO lightweight target detection model to perform obstacle identification on the key elements in the area.

[0118] The real-time obstacle detection framework adopts high-resolution input and deep feature extraction network, and the identified objects include but are not limited to: handwritten annotations, occlusion areas formed by seal covering, fuzzy areas caused by scanning noise, and incomplete areas formed by formula symbol breakage and table frame defects. The output result is a high-precision pixel-level obstacle mask (Mask) and corresponding obstacle type label and confidence score. The obstacle mask is divided into a single-channel mask (only marking the obstacle position, the obstacle area is marked with "1", and the non-obstacle area is marked with "0") or a multi-channel mask (each channel corresponds to one type of obstacle, such as channel 1 marking the occlusion area, channel 2 marking the fuzzy area, and channel 3 marking the incomplete area, and "1" in each channel indicates the corresponding type of obstacle pixel); only the obstacle area with a confidence score greater than or equal to a preset threshold (such as 0.85) is marked, and the mark below the threshold is regarded as a false detection and ignored, and finally the obstacle type (occlusion, blur, and incompleteness) and corresponding position of the key element area are determined.

[0119] S302, for the non-key element area, a simplified detection mode of the YOLO lightweight target detection model is used to identify only the key basic obstacles that may interfere with the analysis of key elements.

[0120] For the non-key element area, a simplified detection mode of the YOLO lightweight target detection model is used for processing efficiency. This mode reduces the input image resolution, reduces the network depth, and only performs rapid screening on key basic obstacles that may cause substantial interference to the analysis of key elements, such as large-area text blur covering, large-scale occlusion across element boundaries, and overall tilt that seriously affects OCR identification. The output result is a low-resolution obstacle existence identifier and a rough position frame.

[0121] Further, the calling of the repair logic corresponding to the obstacle type for targeted processing to generate an obstacle-free element document that preserves the association relationship between each element and the original document coordinates specifically includes:

[0122] S310, for text occlusion or image occlusion, image inpainting logic is adopted to complete the pixels in the occlusion area based on the texture features and color distribution of the surrounding pixels of the occlusion area, and the element integrity is restored.

[0123] In the embodiment of the application, for text occlusion or image occlusion, a deep learning-based image inpainting logic (such as a neural network using Partial Convolution or Gated Convolution) or a traditional image processing algorithm (such as texture synthesis based on Patch Match) can be used to intelligently infer and complete the pixels in the occlusion area based on the texture features and color distribution of the surrounding effective pixels of the occlusion area, and restore the visual integrity of the elements.

[0124] S311, for formula occlusion, the completion is performed by fusing context semantic association and quantum complex word vector features, specifically including:

[0125] S3110, a pre-constructed symbol priori knowledge graph is called to obtain the logical constraints of the occluded symbols.

[0126] The pre-constructed symbol priori knowledge graph stores a large number of logical association rules between mathematical symbols in the form of triples, such as <∑, precedes, variable_sequence>, to provide basic logical constraints for reasoning.

[0127] S3111, a sequence encoder is used to contextually encode the formula symbol sequence before and after the occlusion area to obtain a context encoding result.

[0128] In the embodiment of the application, a bidirectional LSTM or Transformer encoder is used as a sequence encoder to deeply contextually encode the formula symbol sequence within a certain window before and after the occlusion area, capture the long-term dependency relationship and syntax structure between the sequences, and obtain an encoding result vector containing rich context information. For example, for the formula "∑(□+x)" (□ is the occlusion area), the encoder will capture the context features of "∑(" and "+x)", and output a vector containing the position and logical relationship (such as the probability of supplementing the variable "i" after "∑").

[0129] S3112, a logical reasoning decoder is used to combine the knowledge graph rules and the context encoding result to generate a candidate symbol sequence for the occlusion area.

[0130] In the embodiment of the application, a logical reasoning decoder integrated with an attention mechanism is used to dynamically combine the prior rules provided by the knowledge graph and the context encoding result, and autoregressively generate one or more candidate symbol sequences for the occlusion area.

[0131] S3113, the quantum complex word vector is introduced to assign an initial probability to the candidate sequence, and the posterior probability is iteratively optimized through a probability optimization model. When the posterior probability exceeds a preset probability threshold or the number of iterations reaches a preset upper limit, the optimal completion scheme is determined.

[0132] In the embodiments of the present application, the quantum complex word vector is used to encode the deep semantics and structural relationship of mathematical symbols. In this representation system, each mathematical symbol is mapped to a complex vector representation (a+bi), where the modulus represents the intensity or commonness of the symbol, and the phase angle θ (θ=arctan(b / a)) is used to finely depict the combination preference and structural role of the symbol in the mathematical expression, where a is the real part of the complex number, b is the imaginary part of the complex number, and i is the imaginary unit. The compatibility or correlation strength between symbols can be measured by the phase difference Δθ, and the smaller the Δθ, the higher the compatibility in semantics and syntax.

[0133] Specifically, the multiple candidate completion items generated based on the symbol context reasoning model will be assigned an initial probability in combination with the quantum complex word vector: for each candidate symbol (or sequence), calculate the average phase difference Δθ between its quantum complex word vector and the surrounding context symbols. The phase difference is inversely proportional to the initial probability, that is, the smaller the Δθ, the higher the initial probability assigned. This initialization process provides a prior distribution based on semantic and structural consistency for subsequent optimization.

[0134] In the optimization phase, a probability optimization model such as Bayesian optimization or Monte Carlo tree search is used to start with the initial probability, comprehensively consider factors such as the rationality of the completed formula structure and the semantic coherence, and perform multiple rounds of iterative reasoning to dynamically update the posterior probability of each candidate scheme. During the iteration process, the posterior probability converges until the posterior probability of a candidate scheme exceeds a preset threshold (indicating that a high-confidence repair scheme has been obtained), or the total number of iterations reaches a preset upper limit (to prevent infinite loops), and the optimization process terminates. Finally, the candidate scheme with the highest posterior probability is selected as the repair result output.

[0135] S312, for fuzzy obstacles, image denoising and sharpening logic is used to improve the clarity of elements by pixel smoothing denoising and edge enhancement sharpening for text, table or formula fuzziness caused by scanning noise.

[0136] Specifically, for fuzzy obstacles, a denoising and sharpening logic combining traditional image processing and deep learning is used. For text, table or formula fuzziness caused by scanning noise, first, non-local mean denoising (NL-Means) or BM3D algorithm can be used for pixel-level smoothing denoising, and then Laplacian mask or deep learning-based super-resolution network (such as ESPCN) is used for edge enhancement and sharpening to significantly improve the clarity and readability of elements.

[0137] S313, for the incomplete obstacle, call the structure completion logic for formula symbol fracture or table border missing: based on the standard structure of similar elements, combined with the context characteristics of the incomplete area, complete the incomplete part, and restore the complete structure of the element.

[0138] Specifically, for the incomplete obstacle, call the structure completion logic based on the generative adversarial network (GAN) or the structure completion convolutional network for formula symbol fracture or table border missing: the model learns the standard structure characteristics of a large number of similar elements (such as complete mathematical symbols, standard table lines), and combines the local context characteristics of the current incomplete area (such as shape continuity at the fracture, extension direction of the border) to perform generative completion, accurately restoring the complete structure of the element.

[0139] S314, for the detected direction abnormal element (such as a horizontal table, an inclined formula), adopt the principal component analysis-based direction angle detection logic to perform correction.

[0140] In the embodiment of the application, for the detected direction abnormal element (such as a horizontal table, an inclined formula), the principal component analysis (PCA)-based text line direction angle detection technology is adopted to perform automatic correction, and the element direction is standardized, which specifically includes the following steps:

[0141] S3141, according to the coordinate information of the direction abnormal element recorded in the element positioning annotation table, the corresponding area of the element in the document image is cropped, and the coordinate set of all non-background pixel points in the area is extracted.

[0142] Specifically, according to the boundary box coordinates (x1, y1, x2, y2) of the direction abnormal element recorded in the element positioning annotation table, the element area is accurately cropped from the document image; the cropped area is binarized, and the coordinate set P={(xi, yi)} of all foreground (non-background) pixel points in the area is extracted.

[0143] S3142, sample the pixel coordinate set to obtain a sample point set.

[0144] To avoid calculation bottleneck, the coordinate set P is randomly sampled or grid sampled to obtain a representative sample point set P sampled .

[0145] S3143, convert the coordinates of the sample point set into a centralized matrix, wherein each row of the matrix corresponds to the coordinates of a pixel point, and each coordinate value is subtracted by the mean value of the corresponding dimension.

[0146] The coordinates of the sample point set P sampled are organized into an N×2 matrix M (N is the number of sample points), and the mean center of the matrix is calculated to obtain a centralized matrix M centeredi.e. each column element minus the mean of that column.

[0147] S3144, solve the covariance matrix of the centering matrix, decompose the covariance matrix by using matrix decomposition algorithm, get the eigenvalue and eigenvector, wherein the eigenvector direction corresponding to the maximum eigenvalue is the main direction feature vector of the pixel point set.

[0148] Specifically, the centering matrix M centered The 2x2 covariance matrix Cov=(M centered T xM centered ) / (N-1) is calculated; the eigenvalues λ1 and λ2(λ1≥λ2) and the corresponding eigenvectors v1 and v2 are obtained by singular value decomposition (SVD) of the covariance matrix Cov; the direction of the eigenvector v1 corresponding to the maximum eigenvalue λ1 is the main direction of the pixel point distribution.

[0149] S3145, calculate the tilt angle of the direction abnormal element according to the main direction feature vector.

[0150] Specifically, according to the main direction feature vector v1=(u, v), the angle θ (tilt angle) between the direction and the horizontal axis (X axis positive direction) is calculated: θ=atan2(v, u) (radian) or θ=arctan(d / u). Wherein, u is the component of the main direction feature vector on the X axis, and v is the component of the main direction feature vector on the Y axis.

[0151] S3146, take the geometric center of the element region as the rotation origin, and perform inverse rotation transformation according to the tilt angle.

[0152] Specifically, taking the geometric center of the element region as the rotation origin, using affine transformation, inverse (negative angle) rotation transformation is performed according to the calculated tilt angle θ, and the element is corrected back to the horizontal direction.

[0153] S3147, adjust the bounding box of the rotated element region to align the element edges with the coordinate axes, and normalize the pixel values to a preset range.

[0154] Specifically, nearest neighbor or bilinear interpolation is performed on the rotated image region, its bounding box is adjusted, black filling area is eliminated, element edges are aligned with image coordinate axes, and pixel values are normalized to the standard range of [0, 255], and the correction is completed.

[0155] To ensure the integrity and traceability of the processing results, the original coordinates of all key positions of the element are recorded before the rotation processing is performed on the element, so that the original coordinate information of the target element is completely retained. This design allows the processing results to be accurately restored to the original document layout through reverse coordinate transformation after parsing and information extraction, avoiding information loss or misplacement caused by format conversion.

[0156] S400, all key elements in the barrier-free element document are cooperatively encoded to generate a cooperative encoding result; a task token is generated based on the task type identifier and the cooperative encoding result, specific subtasks are split according to the task token and are processed in parallel, the processing precision is adjusted by calling the domain adaptation parameter during processing, and a multi-task parallel processing result set is generated.

[0157] In the embodiment of the application, an improved Vision-Text Transformer (VT-Transformer) is used as a core encoder. In the input layer, the document image is divided into image blocks of a fixed size (such as 16x16 pixels), and is linearly projected into visual tokens (Visual Tokens); at the same time, the text content, formula symbol sequence and table cell content are respectively converted into semantic tokens (Semantic Tokens) through an embedding layer. In order to retain the layout and sequence information of the elements, position embedding (Position Embedding) is added to all tokens, wherein the spatial position embedding is used to encode the two-dimensional coordinates and relative layout of the elements in the document, and the sequence position embedding is used to encode the position of the elements in the reading order.

[0158] In the cooperative attention layer, multi-modal information interaction is realized through a cross-element attention mechanism. This mechanism uses scaled dot-product attention (ScaledDot-ProductAttention) to calculate the correlation between query (Query), key (Key) and value (Value), so that the model can capture deep associations between elements, for example: formula tokens can query and focus on table tokens related to them (such as the formula "C=πd" and the column "diameter d" in the table), and text tokens can query and focus on image tokens described by them (such as the text "as shown in the figure" and the corresponding illustration content). After multiple rounds of interaction, a 512-dimensional fusion feature vector is finally output, which contains the global semantic and structural information of the document elements.

[0159] The decoder adopts a multi-task parallel output architecture, and introduces a task token (TaskToken) as an identifier of different subtasks to distinguish and drive the execution of four subtasks:

[0160] Formula recognition subtask: A Transformer-based decoder is used to map the encoded formula feature sequence to a LaTeX code sequence, while simultaneously outputting the two-dimensional coordinate bounding box of each symbol in the formula in the document.

[0161] Table processing subtask: The attention mechanism is used to decode the table structure, and the coordinates, text content, and tree-like hierarchical relationship (such as parent cell, child cell, and cross-row and cross-column attributes) of each cell in the table are output in parallel to accurately restore the logical structure of complex tables.

[0162] Information extraction subtask: A joint learning framework is used to model entity recognition, relationship extraction, and attribute classification subtasks, and a pre-trained language model is used to enhance cross-modal semantic understanding. Finally, the extracted structured information is output in the form of (subject-relation-object) triples to accurately express the semantic association between elements.

[0163] Content analysis subtask: A graph neural network (GNN) is used to construct a document element association graph, where nodes represent elements of different modalities, and edges represent semantic or reference relationships. The hierarchical layout is used to present the relationship between core elements and auxiliary elements, and the document theme label and visual graph are output to support deep document understanding. To achieve collaborative optimization of multiple tasks, the invention uses the following weighted fusion joint loss function:

[0164] L total =w1×L edit +w2×L IOU +w3×L F1 +w4×L semantic .

[0165] Where: L edit is the formula edit distance loss, which measures the difference between the predicted LaTeX code and the true label using Levenshtein distance, i.e. L edit =Levenshtein(y latex p , y latex ), where y latex p is the predicted LaTeX code sequence, and y latex is the true label sequence.

[0166] L IOU is the table IOU loss, which is obtained by calculating the intersection over union (Intersection over Union) of the predicted cell bounding box and the true bounding box: L IOU =1-(Area(B p ∩B)) / (Area(B p ∪B)), where B pFor the predicted cell bounding box, B is the real bounding box, and Area(·) represents the area calculation function.

[0167] L F1 For the information extraction F1 loss, the F1 value is constructed based on entity recognition and relation extraction: L F1 = 1-F1(entities p , entities; relations p , relations), where entities p , relations p are the predicted entities and relations, and entities, relations are the corresponding real entities and real relations. The real entity is the manually annotated document element, such as diameter d, but the predicted entity may be misjudged as radius r; the real relationship is the association between the manually annotated entities, such as d corresponding to the third column of Table 1, but the predicted relationship may be misjudged as irrelevant.

[0168] L semanti c is the semantic consistency loss, which uses cosine similarity or KL divergence to measure the difference between the predicted element association graph and the manually annotated graph: L semantic = 1-CosSim(G p , G) or L semantic = KL(G p ||G), where G p is the predicted element association graph, G is the real association graph, CosSim() is the cosine similarity calculation function, and KL(||) is the KL divergence calculation function.

[0169] w1 to w4 are weight coefficients, which can be dynamically adjusted according to the document type and task requirements. For example, for formula-intensive documents, w1 can be increased to improve formula parsing accuracy; for documents dominated by tables, w2 and w3 can be increased to optimize table structure and information extraction effect. In an illustrative embodiment, w1 = w3 = 0.3, and w2 = w4 = 0.2.

[0170] Through the above formula, the weight parameters can be dynamically adjusted according to different document types and task requirements (such as for formula-intensive documents, L edit can be increased to maximize the performance of the model.

[0171] S500, performing multi-dimensional verification correction on the multi-task parallel processing result set, integrating the corrected results according to the format requirements in the field adaptation parameters, and forming a final structured document report.

[0172] Further, the following three-dimensional automatic verification and correction are performed on the multi-task parallel processing result set (including LaTeX formula, CSV table, extracted triple, element association graph, etc.):

[0173] (1) Syntax and format verification and correction:

[0174] Formula verification: Syntax analysis is performed on the LaTeX sequence to detect and correct unclosed parentheses (such as {, ()), incorrect operators (such as mistaking \times as x), or unknown environment commands to ensure that they can be compiled by the standard LaTeX engine.

[0175] Table verification: Check the legality of the CSV format and correct format errors caused by cell content containing unescaped commas or line breaks; unify column data types (such as standardizing mixed "date" and "numeric" text in the same column according to domain rules).

[0176] Text verification: Perform spelling check on the text output by OCR recognition and correct obvious misspelled words based on a general dictionary.

[0177] (2) Semantic and consistency verification and correction:

[0178] Term correction: Call the professional term library specified in the domain adaptation parameters to perform consistency verification on professional terms in text and tables. For example, in the medical field, automatically correct the colloquial description "heart attack" to the standard term "myocardial infarction"; in the financial field, unify "YoY growth" to "same period growth rate".

[0179] Association verification: Based on the triples and element association graph generated by information extraction, verify the consistency of cross-element references. For example, if the text mentions "see Table 1", but the "Table 1" node does not exist or the ID does not match in the graph, an alert is triggered and the correct table ID is attempted to be corrected.

[0180] Unit unification: Detect numerical units in tables or text to ensure that units are expressed consistently within the same document (e.g., unify "5kg" and "5000g" to "5kg").

[0181] (3) Logic and business rule verification and correction:

[0182] Value logic verification: In financial reports and other fields, automatically verify whether the "total row" value is equal to the sum of each item, and if there is a deviation, correct or mark it according to the predefined business rules (such as rounding precision).

[0183] Formula derivation verification: In mathematical or engineering documents, check the coherence of the derivation logic of key formulas, or verify whether the formula calculation result matches the related values in the table.

[0184] Cross-reference integrity: Ensure all figures and tables are referenced in the text, and all cited references are present in the reference list.

[0185] Further, according to the predefined format rules in the domain adaptation parameters, the revised parts of the content are integrated into a structured document that conforms to the conventions of the domain:

[0186] Mathematics domain: Organized in the structure of "text + embedded LaTeX formulas + formula explanation + formula index". A unique number is generated for each formula, and it is referenced in the text by number, and a formula index list is generated at the end of the document.

[0187] Finance domain: Organized in the structure of "text analysis summary + structured table + key data visualization + CSV data download link". The system automatically calls the visualization engine to generate line charts, bar charts, etc. according to the table data, and embeds them in the report, while providing a CSV file download link for structured table data.

[0188] Scientific research domain: Organized in the structure of "abstract + text + formula list + table list + references". The formulas and tables in the text are numbered and referenced, and the formula list and table list are generated at the end of the document, and the reference format is automatically adjusted to the required citation format (such as IEEE, APA) according to the adaptation parameters.

[0189] Further, the integrated content is output in multiple interoperable machine-readable and human-readable formats to form the final structured document report:

[0190] Editable Word document (.docx): Retains all styles and formats, making it easy for users to make final review and fine-tuning.

[0191] Standardized PDF document (.pdf): Used for distribution, printing or archiving, keeping the format fixed.

[0192] Digital JSON document (.json): Contains all text content, formula LaTeX code, table CSV data, triple knowledge, element coordinates and association, making it easy for other data analysis systems, knowledge graph construction tools or retrieval platforms to directly call and deeply process.

[0193] The final report retains the coordinate mapping relationship of the elements with the original document, ensuring the traceability of the processing process, and can be directly used for editing, retrieval, data analysis and knowledge discovery, etc. Downstream tasks.

[0194] The embodiment of the present application further provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method of the embodiment of the present application.

[0195] The embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions are used for executing the method of the embodiment of the present application.

[0196] It should be understood that the steps shown above in various forms of flow can be reordered, added or deleted. For example, the steps described in the present application can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions disclosed in the present application can be achieved, which is not limited herein.

[0197] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multimodal document parsing method, characterized in that, The method includes the following steps: S100, perform preprocessing operations on the original document to generate a preprocessed document, and use an object detection model to identify all key elements in the preprocessed document to obtain an element location labeling table; the key elements include at least formulas, tables, text and images, and the element location labeling table contains the element ID, coordinate information and preliminary type corresponding to each key element. S200: Based on the element positioning label table, perform coarse task type identification, and determine the task type identifier of the original document according to the coarse identification result; identify the domain to which the original document belongs based on the metadata of the original document, and generate domain adaptation parameters by combining the task type identifier and the domain to which it belongs; S300, determine the area range of each key element according to the element positioning label table, and perform hierarchical obstacle detection in combination with the task type identifier to obtain the obstacle type of all key elements; The corresponding repair logic is invoked to perform targeted processing and generate an accessible element document that retains the relationship between each element and the original document coordinates. S400, perform co-coding on all key elements in the accessibility element document to generate a co-coding result; Based on the task type identifier and the co-coding result, a task token is generated. Specific sub-tasks are split according to the task token and processed in parallel. During the processing, the domain adaptation parameters are called to adjust the processing precision, and a multi-task parallel processing result set is generated. S500, perform multi-dimensional verification and correction on the result set of the multi-task parallel processing, and integrate the verified and corrected results according to the format requirements in the domain adaptation parameters to form a final structured document report; The process of generating domain adaptation parameters by combining the task type identifier and the domain to which it belongs specifically includes: Based on the identified domain, the initial parameter template for the corresponding domain is called from the pre-built domain parameter template library; The initial parameter template is adjusted according to the task type identifier to obtain the initial weights of the adjusted domain parameter matrix: if the task type identifier is formula processing, the weights of the feature dimensions related to formula symbols in the domain parameter matrix are adjusted, and the knowledge inheritance weights are adjusted to focus on the formula syntax parsing parameters of the domain general model; if the task type identifier is table processing, the weights of the feature dimensions related to table structure in the domain parameter matrix are adjusted, and the precision threshold is adjusted according to the core requirements of the table data. Adjust the number of samples required for meta-learning fine-tuning based on the commonness of the domain: set the number of samples required for common domains to the first preset value, and adjust it to the second preset value for unpopular or scarce domains. At the same time, supplement the domain-specific knowledge graph into the domain knowledge enhancement sub-parameter. The adjusted domain parameter matrix initial weights, meta-learning fine-tuning sample requirements, accuracy thresholds, format rules, and knowledge fusion parameters are integrated into a complete domain adaptation parameter file.

2. The method according to claim 1, characterized in that, The step of performing coarse task type identification based on the element location annotation table, and determining the task type identifier of the original document based on the coarse identification result, specifically includes: The proportion of formula elements in the element positioning label table to the total number of all identifiable elements is calculated. If the proportion is greater than or equal to the first preset proportion, the formula processing task type identifier is activated. Analyze the table elements in the element location label table. If a table element meets any of the following conditions: the proportion of the number of merged cells to the total number of cells in the corresponding table is greater than or equal to the second preset proportion, the number of table rows is greater than the preset number of rows, or the table contains multimodal content, then activate the table processing task type identifier. If the proportion of formula elements to the total number of recognizable elements is less than the first preset proportion, and the table elements do not meet any of the above conditions, then the text processing task type identifier is activated.

3. The method according to claim 2, characterized in that, The metadata of the original document includes at least the title, keywords, author affiliation, document summary, and file attribute information; the identification of the domain to which the original document belongs based on its metadata specifically includes: Perform normalization processing on the metadata of the original document, including word segmentation and stop word removal of text metadata, and uniform standardization mapping of format tags; The domain-general model, pre-trained on multi-domain corpora, is invoked to extract features from the normalized metadata and output the corresponding domain feature vectors. The domain feature vector is matched with the benchmark features of each domain in the preset domain feature library for similarity matching, and the cosine similarity between it and the benchmark feature vector of each domain in the library is calculated to obtain a set of similarity scores. If the maximum similarity score is greater than or equal to the first preset threshold, and the difference between the maximum similarity score and the second maximum similarity score is greater than or equal to the preset difference threshold, then the original document is directly determined to belong to the domain corresponding to the maximum similarity score. If multiple similarity scores are greater than or equal to the second preset threshold, and the difference between these similarity scores is less than the preset difference threshold, then the author's affiliation in the metadata is used for calibration, and a unique domain identifier is finally output.

4. The method according to claim 1, characterized in that, Performing hierarchical obstacle detection in conjunction with the task type identifier includes: performing refined obstacle detection on key element regions that match the task type identifier, and performing basic obstacle detection on non-key element regions; Specifically, refined obstacle detection is performed on key element areas that match the task type identifier, while basic obstacle detection is performed on non-key element areas. This includes: For key element areas that match the task type identifier, a real-time obstacle detection framework based on the YOLO lightweight object detection model is invoked to identify obstacles in key elements within the area. The identified objects include: occluded areas formed by handwritten annotations and stamps, blurred areas caused by scanning noise, and incomplete areas formed by broken formula symbols and missing table borders. For non-key element regions, a simplified detection mode of the YOLO lightweight object detection model is adopted to identify only the key basic obstacles that may interfere with the parsing of key elements.

5. The method according to claim 4, characterized in that, The process involves calling the repair logic corresponding to the obstacle type to perform targeted processing, generating an accessible element document that retains the coordinate association between each element and the original document, specifically including: For text or image occlusion obstacles, image inpainting logic is used to complete the pixels in the occluded area based on the texture features and color distribution of the pixels surrounding the occluded area, thus restoring the integrity of the elements. To address formula occlusion obstacles, we integrate contextual semantic associations with quantum complex word vector features to perform completion. For blurry obstacles, image denoising and sharpening logic is used. For text, tables or formulas that are blurred due to scanning noise, the clarity of elements is improved by pixel smoothing denoising and edge enhancement sharpening. For incomplete or obstructed elements, such as broken formula symbols or missing table borders, the structural completion logic is invoked: based on the standard structure of similar elements, the missing parts are completed by combining the contextual features of the missing area to restore the complete structure of the element. For the detected directional anomalies, a direction angle detection logic based on principal component analysis is used to perform correction.

6. The method according to claim 5, characterized in that, The method for handling formula occlusion obstacles by fusing contextual semantic associations and quantum complex word vector features to perform completion specifically includes: Call the pre-built symbolic prior knowledge graph to obtain the logical constraints of the occluded symbols; A sequence encoder is used to perform context encoding on the sequence of formula symbols before and after the occluded area to obtain the context encoding result; By combining knowledge graph rules and context encoding results with a logical reasoning decoder, candidate symbol sequences for the occluded region are generated; Quantum complex word vectors are introduced to assign initial probabilities to candidate sequences. Then, the posterior probability is iteratively optimized through a probability optimization model. When the posterior probability exceeds a preset probability threshold or the number of iterations reaches a preset upper limit, the optimal completion scheme is determined.

7. The method according to claim 5, characterized in that, For detected directional anomalies, a text line direction angle detection technique based on principal component analysis is used to perform correction, specifically including: Based on the coordinate information of the directionally abnormal elements recorded in the element positioning annotation table, the corresponding region of the element in the document image is cropped out, and the coordinate set of all non-background pixels in the region is extracted. The pixel coordinate set is sampled to obtain the sample point set; The coordinates of the sample point set are converted into a centered matrix, where each row of the matrix corresponds to the coordinates of a pixel, and each coordinate value is subtracted from the mean of the corresponding dimension. The covariance matrix is ​​solved for the centered matrix. The matrix decomposition algorithm is used to decompose the covariance matrix to obtain eigenvalues ​​and eigenvectors. The direction of the eigenvector corresponding to the largest eigenvalue is the main direction eigenvector of the pixel set. The tilt angle of the directional anomaly element is calculated based on the main direction feature vector; Using the geometric center of the element region as the origin of rotation, perform the inverse rotation transformation according to the tilt angle; Adjust the bounding box of the rotated element region to align the element edges with the coordinate axes and normalize the pixel values ​​to a preset range.

8. An electronic device, characterized in that, Including processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 7 by invoking programs or instructions stored in the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 7.

Images