Heterogeneous data analysis and understanding system based on artificial intelligence adaptive feature fusion

Abstract

The embodiment of the invention relates to the technical field of heterogeneous data processing, and discloses a heterogeneous data analysis and understanding system based on adaptive feature fusion. The TFJAN module is used for extracting table structure features and mathematical formula region features in the heterogeneous data through a multi-level table detector and a formula locator; joint feature representation of the table and the formula is obtained; the MSCAM module is used for constructing a spatial dependency relationship of mathematical symbols in a heterogeneous environment, performing context sensing on mathematical symbol features and enhancing the mathematical symbol features; the SCO module is used for carrying out structure unification processing on the heterogeneous data through structure regularization and a structure optimization function; the FSPA module is used for analyzing the structured feature representation of the mathematical formula to obtain the content representation of the mathematical formula; and the FSUN module is used for analyzing semantic information of a mathematical formula through a deep learning model. The technical problem that heterogeneous data formulas are difficult to analyze can be at least solved.

Description

Technical Field

[0001] This application relates to the field of heterogeneous data processing technology, and in particular to a heterogeneous data parsing and understanding system based on artificial intelligence adaptive feature fusion. Background Technology

[0002] Traditional mathematical formula recognition methods largely rely on image recognition technology, typically extracting formulas from documents and then performing separate symbol recognition and parsing. However, this approach has several significant limitations: First, because formulas are often nested within complex table structures, traditional image recognition methods struggle to accurately distinguish between table regions and formula regions, significantly reducing the accuracy of the parsing results. Second, existing mathematical formula recognition technologies primarily focus on symbol recognition, lacking effective mechanisms for handling the relationship between formulas and surrounding tables, semantic understanding of formulas, and their contextual dependencies within the document. Third, the structural complexity of tables themselves also presents a significant challenge to existing technologies, especially in heterogeneous document scenarios where tables and formulas are intertwined. Existing technologies cannot simultaneously process the structured information of both, resulting in unsatisfactory parsing performance.

[0003] To address these issues, researchers in related fields have proposed several deep learning-based methods for parsing tables and formulas. These methods attempt to automatically extract structural information and formula symbols from documents by constructing complex neural network models. However, existing methods still face several challenges: the interaction between tables and formulas is not fully considered when dealing with complex heterogeneous data, and the training and inference processes of the models often struggle to adapt to various document formats and diverse formula structures. Furthermore, effectively fusing the structural information in tables with the symbolic features in mathematical formulas to achieve accurate semantic understanding remains a pressing technical challenge. Summary of the Invention

[0004] One objective of this application is to provide a heterogeneous data parsing and understanding system based on adaptive feature fusion, which at least solves the technical problem of difficulty in parsing formulas for heterogeneous data.

[0005] To achieve the above objectives, some embodiments of this application provide a heterogeneous data parsing and understanding system based on adaptive feature fusion, comprising: a TFJAN module, used to extract table structure features and mathematical formula region features from heterogeneous data through a multi-level table detector and formula locator; employing an adaptive weighting mechanism to perform feature fusion on the table structure features and mathematical formula region features to obtain a joint feature representation of the table and formula; an MSCAM module, used to extract mathematical symbol features from the TFJAN module, process the mathematical symbol features through convolution and pooling layers and a graph attention network, construct the spatial dependency relationship of mathematical symbols in the heterogeneous environment, perform context awareness on the mathematical symbol features, and enhance the mathematical symbol features; an SCO module, used to extract table hierarchical structure features and formula cell features from the heterogeneous data, and perform structure consistency processing on the heterogeneous data through structure regularization and structure optimization functions; an FSPA module, used to capture the structured feature representation of mathematical formulas from the MSCAM module, and parse the structured feature representation of mathematical formulas to obtain the content representation of mathematical formulas; and a FSUN module, used to extract the content representation of mathematical formulas from the FSPA module, and parse the semantic information of mathematical formulas through a deep learning model.

[0006] Compared with related technologies, the solution provided in this application, through an innovative multi-level recognition architecture, deeply mines the inherent relationship between tables and formulas. Utilizing multimodal information fusion and context-aware mechanisms, it overcomes the limitations of traditional methods in heterogeneous data parsing, significantly improving the accuracy and efficiency of joint table and formula parsing. It can automatically identify table structures and mathematical formulas in complex documents and, through a deep semantic understanding model, achieve accurate parsing and semantic reasoning of formula content, thus providing an innovative solution for the field of heterogeneous data parsing. Attached Figure Description

[0007] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0008] Figure 1 This is a schematic diagram of the architecture of a heterogeneous data parsing and understanding system based on adaptive feature fusion, according to an embodiment of this application.

[0009] Figure 2 This is a schematic diagram of the architecture of a TFJAN module according to an embodiment of this application;

[0010] Figure 3 This is a schematic diagram of the architecture of an MSCAM module according to an embodiment of this application;

[0011] Figure 4 This is a schematic diagram of the architecture of an SCO module according to an embodiment of this application;

[0012] Figure 5 This is a schematic diagram of the architecture of an FSPA module provided according to an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The following terms are used in this document:

[0015] Module / Network Abbreviations: TFJAN: Joint Attention Network for Table-Formula Regions; MSCAM: Mathematical Symbol Context Awareness Module; SCO: Structural Consistency Optimization; FSPA: Formula Structure Parsing Attention; FSUN: Formula Semantic Understanding Network; HTSN: Hierarchical Table Structure Analysis Network; FCRM: Formula Cell Recognition Module; MSFE: Multi-Scale Symbolic Feature Extractor; SRRN: Symbolic Relationship Reasoning Network; AFFM: Adaptive Feature Fusion Module; DSMN: Deep Semantic Modeling Network; CEUM: Context Enhancement Understanding Module; SVO: Semantic Validation Optimizer; MLVN: Multi-Level Validation Network; ICM: Intelligent Error Correction Mechanism; AOS: Adaptive Optimization Strategy; MSIF: Multi-Source Information Fuder; CEM: Consistency Evaluation Module; DO: Dynamic Optimizer; ISS: Intelligent Sample Filterer; IUN: Incremental Update Network; PAM: Parameter Adaptation Module; SFA: Scene Feature Analyzer; SOC: Scene Optimization Controller.

[0016] Related technical components / algorithms: BiLSTM: Bidirectional Long Short-Term Memory Network; GNN: Graph Neural Network; CNN: Convolutional Neural Network; MLP: Multilayer Perceptron; GraphSAGE: A graph neural network architecture for large-scale graph representation learning; Channel-wise attention: A feature enhancement technique; TreeAttn: Tree-structured attention mechanism; EdgeAttn: Edge attention mechanism.

[0017] Other important terms: RelationMap: used to establish spatial dependencies between symbols; StructureGate: used to identify special mathematical structural elements; Pool: used for dimensionality reduction and feature extraction; Conv: used for feature extraction; Embedded layer: used to convert symbols into vector representations; Regular: used to prevent model overfitting; Global: used to capture long-range dependencies; Position: used to encode positional information in a sequence.

[0018] This application relates to a heterogeneous data parsing and understanding system based on artificial intelligence adaptive feature fusion.

[0019] like Figure 1 As shown, the system includes:

[0020] The TFJAN module is used to extract table structure features and mathematical formula region features from heterogeneous data through a multi-level table detector and formula locator; an adaptive weighting mechanism is used to fuse the table structure features and mathematical formula region features to obtain a joint feature representation of tables and formulas.

[0021] The TFJAN module extracts table structure features and mathematical formula region features from heterogeneous data through a multi-level table detector (TC_i) and formula locator (FC_j). A multi-layer convolutional neural network (CNN) is used to extract table features layer by layer, with the output dimension of each layer progressively decreasing to accurately capture table structures at different scales. The output feature dimensions of the i-th layer table detector TC_i are 256×256×64, 128×128×128, and 64×64×256, respectively. The formula locator FC_j, designed with a multi-scale feature pyramid, accurately locates mathematical formula regions, supporting the identification of formula regions at different scales in complex documents.

[0022] The TFJAN module employs an adaptive weighting mechanism, which dynamically optimizes the fusion weights of table features and formula region features through backpropagation. The two features are then fused using a weighted fusion mechanism based on channel attention to obtain a joint feature representation of the table and formula.

[0023] The MSCAM module is used to extract mathematical symbol features from the TFJAN module, process the mathematical symbol features through convolution and pooling layers and graph attention networks, construct the spatial dependency relationship of mathematical symbols in heterogeneous environments, perform context awareness on the mathematical symbol features, and enhance the mathematical symbol features.

[0024] The MSCAM module extracts mathematical symbol features from the table and formula regions extracted by the TFJAN module. These features are processed using convolutional and pooling layers. This process employs a multi-layer convolutional network with 64, 128, and 256 channels per layer to progressively extract discriminative features of the symbols in heterogeneous environments. A Graph Attention Network (GAT) is used to establish spatial dependencies among the symbol features, constructing contextual relationships between mathematical symbols in different table and formula environments. This context-aware enhancement of symbol features improves the accuracy of mathematical symbol recognition.

[0025] The SCO module is used to extract table hierarchical structure features and formula cell features from the heterogeneous data, and to perform structural consistency processing on the heterogeneous data through structure regularization and structure optimization functions.

[0026] The SCO module is used to extract hierarchical table features and formula cell features from heterogeneous data and to perform structural consistency optimization. It extracts the hierarchical structure features (H(T)) and formula cell features (FC(C)), and uses structure regularization terms to constrain the structural integrity of heterogeneous tables. The SCO module optimizes the features of table and formula regions through structure optimization functions, ensuring structural consistency during the parsing process and providing a stable structural foundation for subsequent feature enhancement.

[0027] The FSPA module is used to capture the structured feature representation of mathematical formulas from the MSCAM module, and parse the structured feature representation of mathematical formulas to obtain the content representation of mathematical formulas.

[0028] The FFSPA module is responsible for capturing and parsing the structured features of mathematical formulas. Through SoftMax attention computation combined with matrix lookup, key-value (QF·KF^T) mechanisms, this module effectively captures the structural features of mathematical formulas in heterogeneous data. Specifically, based on the symbolic feature output of the MSCAM module, the FSPA module further identifies the hierarchical and structural characteristics of the formulas. Utilizing the structural feature representation of mathematical formulas, the FSPA module parses the formula content and generates a high-quality structured feature representation, providing a solid foundation for subsequent content understanding.

[0029] The FSUN module is used to extract the content representation of the mathematical formula from the FSPA module and parse the semantic information of the mathematical formula through a deep learning model.

[0030] The FSUN module extracts the content representation of mathematical formulas from the FSPA module and further parses the semantic information of the mathematical formulas through a deep learning model. It obtains the content representation of mathematical formulas from the FSPA module and uses it as input for further semantic analysis. Through deep neural networks (such as LSTM and Transformer), the FSUN module performs semantic parsing of mathematical formulas, understands the mathematical concepts and relationships involved in the formulas, and ultimately generates the semantic representation of the formulas, providing efficient knowledge graphs or other semantic understanding frameworks for subsequent applications.

[0031] The system receives heterogeneous data input, typically complex documents containing tables and mathematical formulas. The TFJAN module extracts regional features from the tables and formulas, the MSCAM module extracts mathematical symbol features and performs context awareness, the SCO module optimizes structural consistency, the FSPA module parses the formula structure, and the FSUN module performs semantic understanding. The system outputs the parsed mathematical formulas and table content, generating structured parsing results for subsequent applications (such as automated document generation and intelligent reasoning). Through the collaborative work of these modules, this embodiment achieves an efficient and accurate heterogeneous data parsing and understanding system, which can be widely applied in fields such as academic literature analysis, automated data extraction, and intelligent education.

[0032] In the technical solutions of each module of this application, the variable symbols (such as α, β, γ, λ, etc.) only have clear meanings within the specific technical context of their respective modules. Variables with the same name in different modules represent different adjustment factors, weight parameters, or functional parameters, and there is no overlap between them. The naming of the variables mentioned above is for simplification only. The technical solutions of each module ensure that the algorithm logic and parameter adjustment mechanisms of different modules are independent through strict division of variable scope, avoiding ambiguity caused by symbol reuse. Those skilled in the art can clearly understand the unique physical meaning and functional definition of a variable based on the specific module and context in which it resides.

[0033] Furthermore, the TFJAN module includes: a table feature extraction unit, which extracts table structure features through a multi-level convolutional network; a formula feature extraction unit, which extracts mathematical formula region features through a multi-scale feature pyramid; and an adaptive weighting unit, which dynamically optimizes the fusion weights of the table structure features and mathematical formula features through backpropagation.

[0034] Furthermore, such as Figure 2As shown, the TFJAN module includes: TF(D)=∑[α_i·TC_i(D)]*∑[β_j·FC_j(D)]; where: D represents heterogeneous data with a dimension of 1024×1024×3; α_i, β_j are adaptive weights; TC_i is the i-th layer table detector, i∈[1,3], with output feature dimensions of 256×256×64, 128×128×128, and 64×64×256 respectively, extracted through a multi-layer convolutional network; FC_j is the j-th layer formula locator, j∈[1,4], extracting mathematical formula region features through a scale feature pyramid; the symbol * represents feature fusion based on channel-wise attention.

[0035] The TFJAN module, through multi-level feature extraction and adaptive weighting mechanisms, achieves the fusion of table structure features and mathematical formula region features, thus providing efficient feature representation for intelligent parsing of heterogeneous data. Input data D:

[0036] The input data D is a three-dimensional matrix of size 1024×1024×3. This dimension is specifically designed to meet the multi-scale feature capture requirements of heterogeneous documents, typically color images or multi-channel heterogeneous document data. This input data contains mixed information from table regions and mathematical formula regions.

[0037] Table Feature Extraction (TC_i): The TFJAN module extracts structured features of tables through a multi-layered table detector (TC_i). The table detector TC_i consists of three convolutional networks of different layers:

[0038] TC_1: The first-layer table detector processes the input data using convolution operations, outputting a feature dimension of 256×256×64. TC_2: The second-layer table detector further processes the output of TC_1, outputting a feature dimension of 128×128×128. TC_3: The third-layer table detector continues to process the output of TC_2, outputting a feature dimension of 64×64×256. Each table detector layer employs layer-by-layer convolution operations to extract table structure features at different scales, ensuring the capture of detailed table information from different levels.

[0039] Formula Feature Extraction (FC_j): Simultaneously, the TFJAN module extracts features from the mathematical formula region using a multi-layered formula locator (FC_j). The formula locator FC_j employs a scale feature pyramid design to ensure accurate feature extraction of the mathematical formula region at different scales. The formula locator consists of four different layers:

[0040] FC_1 to FC_4: FC_1 undergoes multi-scale feature pyramid processing, and then FC_2 / 3 / 4 undergo layer-by-layer feature extraction. These formula locators extract feature information of the mathematical formula region layer by layer, ensuring that the complete form of the formula is captured at different scales.

[0041] For the fusion of table features and formula region features, the TFJAN module employs an adaptive weighting mechanism. Weights α_i and β_j are used to control the fusion strength of table features and formula features, respectively, and are dynamically adjusted through a backpropagation algorithm. This mechanism enables the system to automatically adjust the feature fusion strategy in different document scenarios, ensuring the optimal combination of table structure features and mathematical formula region features.

[0042] α_i: Adaptive weights, controlling the importance of fusing table features at different levels. β_j: Adaptive weights, controlling the importance of fusing formula region features at different levels.

[0043] Feature Fusion: Finally, the TFJAN module uses a channel-wise attention-based feature fusion strategy to weight and fuse table features and formula features. Specifically, by weighting the features of each channel, important features are given a larger proportion in the final fusion result.

[0044] The input heterogeneous data D is processed by a three-layer table detector TC_i and a four-layer formula locator FC_j. Each layer of the table detector and formula locator uses convolution operations to extract features at different scales, and finally outputs table features and formula region features TF_D.

[0045] After extracting table features and formula region features, the system automatically calculates adaptive weights α_i and β_j based on the different types and complexities of the input data, and optimizes the weights through a backpropagation algorithm. In this way, the system can dynamically adjust the feature fusion strategy according to the relative importance of tables and formulas in the document.

[0046] A feature fusion method based on channel attention is used to weight and combine table features and formula region features to obtain a joint feature representation of the table and formula. This representation will simultaneously contain features of the table structure and mathematical formulas, providing richer input information for subsequent modules (such as MSCAM, SCO, FSPA, etc.).

[0047] Ultimately, the TFJAN module outputs a joint feature representation that integrates table and formula features. This representation contains structural and semantic information about the table and formula regions in the document, providing accurate foundational data for subsequent feature enhancement and semantic understanding modules.

[0048] Through the above implementation, the TFJAN module in this embodiment can simultaneously process tables and mathematical formulas in heterogeneous data, automatically perform feature fusion, and effectively extract comprehensive information from table structure and formula regions. By employing multi-layer convolutional networks and scale feature pyramids, it can accurately capture table and formula information at different scales. An adaptive weighting mechanism enables the system to dynamically adjust the fusion method of table and formula region features according to the characteristics of different documents. A channel-based attention mechanism ensures that the most important features are strengthened, improving the model's ability to process complex documents. This significantly improves the joint parsing accuracy of tables and formulas in heterogeneous data, providing strong support for subsequent intelligent parsing and understanding.

[0049] Furthermore, such as Figure 3 As shown, the MSCAM module includes: M(S)=σ(Conv(Pool(S)))*S+γ·RelationMap(S); where: S is the symbolic feature tensor obtained from the TFJAN module, batch_size=32; this feature tensor is specifically used to process mathematical symbol sequences in heterogeneous data.

[0050] Conv uses 3×3 convolutional kernels, stacked in three layers, with 64, 128, and 256 channels respectively, to extract discriminative features of symbols in heterogeneous environments layer by layer; Pool output has a fixed dimension of 7×7; γ is dynamically adjusted within the range of [0.1, 0.9]. RelationMap is based on a graph attention network with a neighborhood size of K=8, establishing spatial dependencies of symbols in heterogeneous environments.

[0051] The main function of the MSCAM module is to extract symbolic features from the TFJAN module and process them through convolutional operations, pooling layers, graph attention networks, and other methods to capture the spatial dependencies of symbols in heterogeneous environments and enhance the context-awareness of symbolic features.

[0052] The input S is a symbolic feature tensor obtained from the TFJAN module, with a batch size of 32, specifically designed to handle mathematical symbols in heterogeneous data. Convolutional operations are applied to the input symbolic feature tensor S to extract discriminative features of the symbols. This process uses three 3×3 convolutional kernels, stacked in a three-layer convolutional network with the following channel counts: first layer: 64 channels; second layer: 128 channels; third layer: 256 channels. Each convolutional operation progressively extracts multi-level information of the symbolic features and enhances its discriminative ability in heterogeneous environments.

[0053] Pooling is performed on the features output by convolution, using a 7×7 pooling layer for downsampling to ensure the structural integrity of the symbols and feature consistency in heterogeneous environments. Pooling effectively reduces the dimensionality of features, improves the computational efficiency of the model, and preserves important feature information.

[0054] After convolution and pooling operations, a sigmoid activation function is used to perform a non-linear transformation on the output, enhancing the network's expressive power. This step helps activate important information in the features and suppress irrelevant or redundant information.

[0055] The convolutional output and the symbolic feature tensor S are weighted and fused using a channel attention mechanism (*symbols represent channel-wise attention fusion). This uses attention weighting based on the convolutional results to ensure that the most important features are enhanced, thereby improving the symbolic representation capability.

[0056] The RelationMap in the Graph Attention Network (GAT) is used to further model the spatial dependencies of symbolic features. Specifically, the RelationMap uses a neighborhood size of K=8 to capture the relationships between symbols, establish the spatial dependencies of symbols in heterogeneous environments, and enhance the context awareness of symbols.

[0057] The adjustment coefficient γ is an adaptive adjustment parameter that controls the weighted balance between convolution and attention mechanisms. The value of γ ranges from [0.1, 0.9]. Its dynamic adjustment can find the optimal balance between local features and global relationships, further improving the robustness and recognition accuracy of symbolic features in heterogeneous scenarios.

[0058] Based on the symbolic features obtained from convolution and pooling, the output of the RelationMap module is fused using a channel attention mechanism to obtain the enhanced symbolic features M_S. The enhanced symbolic features are then output to provide accurate symbolic representations for subsequent modules (such as the SCO and FSPA modules).

[0059] Furthermore, such as Figure 4 As shown, the SCO module includes: S(T) = Optimize(H(T), FC(C)) + λ1·Regular(T); where: H(T) is the table hierarchical structure feature; FC(C) is the formula cell feature; Regular is the structure regularization term, which constrains the structure of heterogeneous tables; λ1 is the balance factor, ranging from [0.3, 0.7], which dynamically adjusts the weights of structure optimization and normalization; Optimize is the structure optimization function.

[0060] First, hierarchical structure features H_T of the table are extracted from the input heterogeneous data using a deep convolutional neural network or other suitable feature extraction methods. These hierarchical features reflect the overall structure and hierarchical relationships of the table, ensuring consistency between its various parts. For example, H_T may contain information about the table's rows, columns, cells, and their nesting relationships.

[0061] Next, the module uses a specialized formula cell detection algorithm to extract the FC_C feature of cells containing formulas in heterogeneous data. Formula cell features identify cells with special formats (such as those containing mathematical formulas, charts, special symbols, etc.) in the table. Through a deep learning model, these features can be automatically identified and extracted, ensuring the system can handle various complex formatted content in tables.

[0062] The `Optimize()` function uses a structure optimization function to process the extracted table hierarchy features (H_T) and formula cell features (FC_C). The goal of the structure optimization function is to ensure that the relationships between cells, rows, and columns in the table conform to a standard structure, and to correctly embed special formatting elements such as formula cells into the overall table structure. The optimization process includes adjusting the layout of rows and columns, merging and splitting cells, etc., to achieve structural consistency in the table.

[0063] To avoid unreasonable table structures (such as disordered rows and columns, duplicate cells, etc.) during the optimization process, the SCO module introduces a structure regularization term. This regularization term constrains the integrity and rationality of the table structure, ensuring that the final output table conforms to actual format specifications. This regularization process uses learned rules or experience to constrain the model's output, preventing non-standard results.

[0064] The balancing factor λ1 is used to adjust the weights between structure optimization and regularization terms during the optimization process. Specifically, the choice of λ1 value affects the relative importance of structure optimization and regularization. This value ranges from [0.3, 0.7]. Through dynamic adjustment, the SCO module can adapt to the structural characteristics of tables in different documents. For example, in some documents, the consistency of table structure may be more important than regularization constraints, while in other documents, stronger structure normalization may be required.

[0065] Finally, the SCO module outputs a globally consistent and standardized table structure by fusing the optimized table structure with the features of formula cells. This optimized table structure provides clearer input for subsequent parsing modules (such as the FSPA and FSUN modules), facilitating further formula parsing and semantic understanding.

[0066] Furthermore, such as Figure 5As shown, the FSPA module includes: FS(F) = SoftMax(QF·KF^T / √d)·VF + λ2·StructureGate(F); where: QF, KF, and VF are constructed based on MSCAM output features, with a dimension of 256, and are used to capture the structured feature representation of mathematical formulas in heterogeneous data; d is the feature dimension, used for normalization; StructureGate identifies special structural elements and processes complex mathematical expressions in heterogeneous data; λ2 adaptively adjusts the range [0.2, 0.8] to dynamically balance general features and special structural features.

[0067] The FSPA module receives the output formula features F from the MSCAM module and constructs three matrices—QF (Query matrix), KF (Key matrix), and VF (Value matrix)—based on these symbolic features. These matrices have a dimension of 256 and correspond to the query (Q), key (K), and value (V) matrices of the formula, respectively, aiming to capture multi-dimensional information about the formula structure.

[0068] Using the QF, KF, and VF matrices, the FSPA module extracts attention features from symbolic features through an attention computation layer. Next, the SoftMax function is applied to calculate the attention weights of the formula. SoftMax, by weighting the normalized features, efficiently captures key elements in the formula structure, giving greater attention to the core features of the formula.

[0069] For certain complex mathematical formulas (such as those containing special structures like integrals, matrices, and square roots), the FSPA module introduces the StructureGate mechanism. The function of StructureGate is to identify and process these complex structural elements. By dynamically adjusting λ² (range [0.2, 0.8]), the structure gating mechanism can balance general features with special structural features, ensuring that special mathematical structures are fully identified and processed.

[0070] λ² is an adaptive adjustment factor used to balance the weights of general features and specific structural features. The value of λ² is dynamically adjusted between [0.2, 0.8], which can be adjusted according to the complexity of different formulas. For simple formulas, a smaller value of λ² is used to focus on general features; while for complex formulas, a larger value of λ² is used to highlight the processing of specific structural features.

[0071] Ultimately, the FSPA module outputs a representation FS_F that contains the structured features of the formula. This representation captures the basic structure of the formula, the relationships between symbols, and the processing results of complex structural elements. FS(F) will provide a precise representation of the formula content for subsequent modules (such as the FSUN module), supporting semantic understanding and reasoning of the formula.

[0072] Furthermore, the FSUN module includes: U(P,C)=Parse(P)+α·Context(C)+β·Logic(P,C); where: P is the formula content representation extracted by the FSPA module, carrying the core semantic information of mathematical formulas in heterogeneous data; C is the table context information, from the TFJAN module, used to establish the semantic association between formulas and table structures; α and β are dynamic weights, ranging from [0,1], adaptively adjusting semantic parsing and context understanding; Parse, Context, and Logic are implemented using deep neural networks to construct a multi-level semantic understanding framework for heterogeneous data.

[0073] The formula content P represents the mathematical formula features extracted from the FSPA module, containing the basic structure and symbolic information of the formula. Through symbolic sequences and structured features, P carries the core semantic information of the formula, serving as input for subsequent semantic parsing.

[0074] The table context information C comes from the TFJAN module and primarily provides information about the table structure within the document. Combined with the formula content P, C helps establish a semantic connection between the formula and the table structure. For example, table headings, column names, or row names can help understand the meaning of a formula, especially when some formulas depend on table data.

[0075] The formula semantic parsing (Parse) module uses a deep neural network to perform basic semantic parsing of the formula content. The purpose of this module is to understand the symbols in the formula and their relationships; for example, parsing "x...". 2 +y 2 =r 2 The Parse module extracts the basic semantic information of a formula by embedding symbols (e.g., through word vectors) and parsing graphs (e.g., the tree structure of the formula).

[0076] Contextual understanding uses contextual information (C) from the table to understand the formula within that context. This contextual information might include relevant column or row headings in the table, or the semantics of the cell containing the formula. By combining this information with the table's structure, the Context module helps understand the meaning and function of the formula within that specific table. For example, in scientific papers, formulas might depend on certain variables or data in the table, making contextual information crucial.

[0077] The Logic module performs logical reasoning by combining the semantic information P of the formula with the contextual information C. Its task is to identify potential relationships within the formula and deduce its rationality and semantic consistency. For example, in complex mathematical formulas, there may be mathematical relationships between symbols (such as equality, inequality, causality, etc.). The Logic module will combine the contextual information C to perform logical reasoning to ensure that the reasoning process conforms to logical rules.

[0078] The weighting coefficients α and β control the importance of the Parse, Context, and Logic modules in the final formula comprehension. By dynamically adjusting the values ​​of α and β, the FSUN module can automatically balance the contributions of basic formula analysis, contextual understanding, and logical reasoning based on different formula types and contexts. For example, when the formula is complex, the Logic module may have a larger weight, while in simple mathematical formulas, the Parse module may have a dominant weight.

[0079] Ultimately, the FSUN module generates a semantic representation U of the formula by weightedly fusing the outputs of Parse, Context, and Logic. U not only contains the basic semantic information of the formula but also combines the table context and the results of logical reasoning, providing a complete and accurate understanding of the formula.

[0080] The FSUN module employs three deep neural network modules: Parse, Context, and Logic, to perform semantic modeling of formulas at three levels: basic parsing, contextual understanding, and logical reasoning, respectively. This multi-layered semantic understanding framework enables the system to more comprehensively understand the meaning of formulas. Dynamically adjustable weight coefficients α and β automatically adjust the contribution of each module based on the characteristics of formulas in different documents. This adaptive feature allows the FSUN module to flexibly respond to different application scenarios, thereby improving the accuracy of formula understanding.

[0081] The FSUN module tightly integrates formula content with table context information, especially when the formula depends on data or variables in the table, enhancing the semantic understanding of the formula by leveraging the table context. Through the logical reasoning of the Logic module, FSUN can not only identify symbols in the formula but also understand the relationships between them, ensuring the semantic consistency of the formula. Especially in understanding complex formulas, logical reasoning ensures that the derivation process and results are reasonable.

[0082] Furthermore, the system also includes: a multi-level verification network for performing multi-level verification on the parsed formula structure to ensure the syntactic, semantic, and logical rationality of the formula; an intelligent error correction mechanism for correcting parsing errors based on historical error correction records and similar case references; and an adaptive optimization strategy for optimizing the parsing process based on optimization functions and dynamic weight adjustment rules.

[0083] Furthermore, the system also includes: a multi-source information fusion unit, used to fuse table features, structural analysis results, symbol recognition results, semantic parsing results, and verification and error correction results, and to perform deep combination according to dynamic fusion weights; a consistency evaluation module, used to evaluate the accuracy of the fusion results and calculate the similarity between the parsing results and the standard template library; and a dynamic optimizer, used to optimize the parsing results and adjust the optimization weights through temporal features.

[0084] Furthermore, the system also includes: an intelligent sample filter for evaluating the quality of newly input table formula samples and filtering them based on sample quality, diversity, and difficulty; an incremental update network for dynamically updating the system model, learning new data patterns based on new samples, and simultaneously performing a selective forgetting mechanism to optimize the storage structure; and a parameter adaptation module for performing parameter adjustments and preventing overfitting through regularization control.

[0085] The system achieves intelligent processing of heterogeneous data through innovative architecture design. At the feature processing level, a joint attention mechanism for table-formula regions is implemented through TF(D), combined with M(S) to complete context awareness of symbols, and structural consistency is optimized based on S(T). The system supports input processing capabilities of 1024×1024×3 dimensions, achieves accurate formula localization through multi-scale feature pyramids, and introduces a dynamic γ parameter in the range of [0.1, 0.9] to ensure flexible adaptability of the processing.

[0086] The system makes deep use of the table features T_feat output by the TFJAN module. In the processing flow, the system inherits the 1024×1024×3 input dimension design of claim 1 and performs deep structured parsing of the TC_i features (i∈[1,3]). The processing result of this module provides necessary table structure information, outputs cell-level semantic representations, and supports the verification mechanism of the SCO module, specifically including:

[0087] Hierarchical Table Structure Analysis Network (HTSN): H(T) = TreeAttn(Cell(T)) + EdgeAttn(Border(T)); where: T is the table feature extracted by T_feat in the TFJAN module, containing multi-dimensional structural information of tables in heterogeneous data; Cell is the cell positioning function, which enables accurate positioning of cells containing formulas in heterogeneous scenarios; Border is the table border detection function, which solves the problem of identifying the integrity of table structure in heterogeneous data; TreeAttn ​​constructs a hierarchical semantic representation of heterogeneous tables through a tree structure attention mechanism; EdgeAttn ​​handles the complex spatial relationships of cells in heterogeneous scenarios through an edge attention mechanism.

[0088] Formula Cell Recognition Module (FCRM): Where: C is the cell feature set, which is specifically used to identify special cells in heterogeneous tables that contain mathematical formulas; The convolutional feature extractor, with a 3-layer structure {64, 128, 256}, adaptively extracts visual features from formula cells; ψ is a graph feature aggregator with a neighborhood range of K=6, establishing semantic dependencies between cells; R is a relation modeling function, solving the contextual understanding problem of formula cells in heterogeneous tables; GNN adopts a 3-layer GraphSAGE structure to enhance the structured feature representation in heterogeneous scenarios.

[0089] Through an innovative hierarchical analysis method, accurate parsing of complex tables is achieved. Specifically, the H(T) function establishes a hierarchical representation of the table using the TreeAttn ​​mechanism, and FC(C)... Two processors, ψ and K=6 (with dimensions {64, 128, 256} and K=6 respectively), accurately identified the formula cells. Through the introduction of the EdgeAttn ​​mechanism, the system can effectively handle complex spatial relationships in heterogeneous scenarios, providing reliable structured support for subsequent symbol recognition and semantic understanding.

[0090] The system performs in-depth processing based on the symbolic features M(S) output by the MSCAM module, while fully utilizing the provided table structure information. Specifically, the system inherits the batch_size=32 design of symbolic features in MSCAM and achieves adaptive fusion of convolutional kernels {3,5,7} at different scales through the multi-scale feature extractor MSFE. It is worth noting that the recognition results of this module directly serve the semantic parsing process and provide basic data support for the verification mechanism, specifically including:

[0091] Multi-Scale Symbolic Feature Extractor (MSFE): F(S) = concat[λ_i·Conv_i(S)] + β·Global(S); where: S is the mathematical symbol feature map, specifically for handling multi-scale representations of mathematical symbols in heterogeneous tables; Conv_i uses convolutional kernels of different scales {3, 5, 7} to adapt to mathematical symbols of different sizes and complexities in heterogeneous scenarios; λ_i is the scale weight, which achieves adaptive fusion of heterogeneous symbol features through dynamic learning; Global is the global context feature, capturing long-range dependencies of symbols in heterogeneous data; β is the global information weight, ranging from [0.1, 0.4], balancing local details and global semantics.

[0092] Symbolic Relational Reasoning Network (SRRN): R(F) = GNN(V(F), E(F)) + μ·Position(F); where: F is the feature output by MSFE, containing multi-dimensional information of mathematical symbols in heterogeneous scenarios; V is the symbol node feature, constructing a graph structure representation of mathematical symbols in heterogeneous data; E is the relation edge between symbols, modeling the semantic association between symbols in heterogeneous environments; Position is the position encoding information, solving the spatial dependency problem of symbols in heterogeneous scenarios; μ is the position weight, ranging from [0.2, 0.6], dynamically adjusting the importance of spatial information.

[0093] The Adaptive Feature Fusion Module (AFFM) is defined as follows: A(F,R) = σ(MLP([F;R]))*W(F) + γ·R; where: F is the symbolic feature, which carries the basic representation of mathematical symbols in heterogeneous data; R is the relational feature, which reflects the structured association between symbols in heterogeneous scenarios; MLP adopts a three-layer structure of {256,128,64} to achieve deep feature fusion; W is the feature transformation matrix, which enhances the expressive power of heterogeneous features; and γ is the fusion coefficient, ranging from [0.3,0.7], which adaptively adjusts the feature fusion intensity.

[0094] Accurate recognition of mathematical symbols in a tabular environment was achieved using the F(S) feature extraction function and the R(F) relational reasoning network. Specifically, global features were dynamically balanced with local features through β weights (range [0.1, 0.4]), while position encoding ensured effective utilization of spatial information through μ weights (range [0.2, 0.6]). In particular, deep fusion and optimization of symbolic features were achieved through the MLP structure {256, 128, 64} and fusion coefficient γ (range [0.3, 0.7]) in the AFFM module.

[0095] Building upon the results of mathematical symbol recognition and fully utilizing the structural parsing capabilities of the FSPA module, the system transforms the recognized symbol sequence X into a 128-dimensional embedding representation through a deep semantic modeling network (DSMN). Combined with the structured features FS(F) provided by FSPA, it achieves complete semantic understanding. The parsing results of this module provide the necessary semantic foundation for verification and error correction, specifically including:

[0096] Deep Semantic Modeling Network (DSMN): L(X) = BiLSTM(Emb(X)) + α·Tree(X); where: X is the sequence of recognized symbols, which is the basic semantic unit of mathematical formulas in heterogeneous data; Emb is the symbol embedding layer (128 dimensions), which constructs the semantic representation space of mathematical symbols in heterogeneous scenarios; BiLSTM captures the bidirectional contextual dependencies of mathematical formulas in heterogeneous data; Tree constructs syntax tree features to parse the hierarchical structure of complex mathematical expressions in heterogeneous environments; α is the tree structure weight, ranging from [0.4, 0.8], which balances the importance of sequence features and structural features.

[0097] Context Enhancement Understanding Module (CEUM): C(L,T)=Fusion(L,TableContext(T))*Attention(L); where: L is the semantic feature, representing the initial semantic representation of the formula in heterogeneous data; T is the table context, providing structured environmental information in heterogeneous scenarios; TableContext specifically extracts the contextual semantics of the table to establish the association between the formula and the table; Fusion realizes the deep fusion of heterogeneous features to enhance the contextual understanding ability of the formula; Attention dynamically focuses on key semantic information to improve the accuracy of understanding heterogeneous scenarios.

[0098] Semantic Validation Optimizer (SVO): V(C) = Verify(Rules(C)) + β·Probability(C); where: C is the context-enhanced feature, containing complete semantic information of formulas in heterogeneous data; Rules is the set of semantic rules, ensuring the rationality of formula parsing in heterogeneous scenarios; Verify performs rule verification, ensuring the accuracy of heterogeneous data parsing; Probability evaluates the credibility of semantic understanding, enhancing system robustness; β is the probability weight, ranging from [0.2, 0.6], dynamically balancing rule constraints and probability evaluation.

[0099] By employing the L(X) semantic modeling function and the C(L,T) context enhancement module, accurate parsing of complex mathematical formulas in tables is achieved. Specifically, through a combination of BiLSTM and Tree structures, along with dynamic weights α (range [0.4, 0.8]), the system effectively fuses sequence features and structural features. The V(C) validation optimizer and its probability weights β (range [0.2, 0.6]) further ensure the accuracy and reliability of semantic parsing.

[0100] Based on the semantic parsing results, and deeply integrated with the semantic understanding capabilities of the FSUN module, the system comprehensively validates the output formula parsing result P through a multi-level validation network MLVN. Simultaneously, it utilizes the semantic understanding framework U(P,C) provided by FSUN to support intelligent error correction. The processing results of this module provide quality assurance for result integration, specifically including:

[0101] Multilevel Validation Network (MLVN): V(P)=∑[w_i·Check_i(P)]·Confidence(P); where: P is the formula parsing result, carrying the complete information of complex formulas in heterogeneous data; Check_i designs a three-layer validator (syntax, semantics, logic) to fully guarantee the parsing quality in heterogeneous scenarios; w_i is the validation weight, dynamically adjusting the importance of different levels of validation; Confidence evaluates the confidence of heterogeneous data parsing; i∈[1,3] represents the validation level, constructing a complete heterogeneous data validation system.

[0102] Intelligent Error Correction Mechanism (ICM): C(E) = Correct(E) + λ·History(E) + μ·Similar(E); where: E is the detected error item, specifically identifying various error types in the heterogeneous data parsing process; Correct is the basic error correction function, providing correction schemes for common errors in heterogeneous scenarios; History is based on historical error correction records, enhancing the experience learning ability of heterogeneous data processing; Similar enhances the system's adaptability to new heterogeneous data through similar case references; λ and μ are weighting factors, both ranging from [0.2, 0.8], dynamically balancing multi-source error correction information.

[0103] Adaptive Optimization Strategy (AOS): O(P,C) = Update(P,C) * Filter(Threshold(P,C)); where: P is the original formula, containing the mathematical expression to be optimized in heterogeneous data; C is the error correction result, reflecting the correction information under heterogeneous scenarios; Update performs intelligent updates to continuously optimize the processing capability of heterogeneous data; Filter is based on threshold filtering to ensure the quality standard of heterogeneous data processing; Threshold dynamically adjusts the threshold to adapt to heterogeneous scenarios of different complexities.

[0104] The validation and error correction of tabular formulas are achieved through the V(P) validation function and the C(E) error correction mechanism. The three-layer validator {syntax, semantics, and logic} provides comprehensive quality assurance through dynamic weights w_i, while the History and Similar functions, combined with weight factors λ3 and μ (both ranging from [0.2 to 0.8]), establish a complete error correction system. In particular, the O(P,C) optimization strategy ensures processing quality in heterogeneous scenarios through dynamic threshold adjustment.

[0105] Integrating the above processing results, a unified result processing framework was constructed using the Multi-Source Information Fusion (MSIF) fusion engine. Specifically, the system receives table features, structural analysis results, symbol recognition results, semantic parsing results, and verification and error correction results. These results are expressed using the R_i feature set and intelligently combined through dynamic fusion weights α_i. Simultaneously, the processing results of this module provide basic data support for online learning, specifically including:

[0106] Multi-Source Information Fusion (MSIF): F(R) = Combine(∑[α_i·R_i]) + β·Cross(R); where: R_i is the recognition result of each module, unifying the multi-dimensional output of heterogeneous data processing; α_i is the dynamic fusion weight, adaptively adjusting the contribution of different modules; Combine realizes deep feature combination, improving the overall understanding of heterogeneous data; Cross performs cross-validation, ensuring the consistency of results in heterogeneous scenarios; β is the cross weight, ranging from [0.3, 0.7], balancing the main features and validation features.

[0107] Consistency Evaluation Module (CEM): E(F) = Evaluate(F) + γ·Distance(F,G); where: F is the fusion result, reflecting the comprehensive output of heterogeneous data parsing; G is the standard template library, providing standard references for heterogeneous scenarios; Evaluate performs feature evaluation to quantify the quality of heterogeneous data processing; Distance calculates the similarity with the standard template, providing objective evaluation basis; γ is a balancing factor, ranging from [0.4, 0.6], adjusting the weight distribution of evaluation dimensions.

[0108] Dynamic Optimizer (DO): D(E) = Optimize(E) * Weight(Time(E)); where: E is the evaluation result, carrying the real-time performance index of heterogeneous data processing; Optimize performs performance optimization to improve the efficiency of heterogeneous data processing; Weight dynamically adjusts the weights based on time-series information to adapt to the processing rhythm of heterogeneous data; Time reflects the processing time-series characteristics to ensure real-time response capability in heterogeneous scenarios.

[0109] The optimal combination of processing results was achieved through the F(R) fusion function and the E(F) evaluation function. Specifically, the Combine operation, combined with Cross-validation, achieved a balance between principal and validation features through β weights (range [0.3, 0.7]). In particular, an objective evaluation system was established through Distance(F,G) similarity calculation and γ balancing factors (range [0.4, 0.6]). The dynamic optimizer D(E) ensured the system's real-time response capability through temporal weight adjustments.

[0110] Based on the constructed complete recognition framework, the quality of newly input table formula samples X is evaluated through the Intelligent Sample Filter (ISS). The system inherits the processing capabilities of the preceding claims, particularly the result evaluation mechanism E(F), and achieves dynamic model optimization through the Incremental Update Network (IUN). The learning outcomes of this module directly affect the scene adaptation effect, specifically including:

[0111] Intelligent Sample Filter (ISS): S(X) = Quality(X) * Diversity(X) + λ * Difficulty(X); where: X is the newly input table formula sample, representing the incremental learning object of heterogeneous data; Quality evaluates the sample quality to ensure the effectiveness of learning from heterogeneous data; Diversity measures the sample diversity to expand the coverage of heterogeneous scenarios; Difficulty evaluates the sample difficulty to balance the learning gradient of heterogeneous data; λ is the difficulty weight, ranging from [0.2, 0.6], which adjusts the degree of influence of learning difficulty.

[0112] Incremental Update Network (IUN): U(M,X)=Base(M)+α·New(X)+β·Forget(M); where: M is the existing model parameters, maintaining the basic ability to process heterogeneous data; X is the selected new sample, providing incremental knowledge of heterogeneous scenarios; Base maintains the basic performance of the model, ensuring the stability of heterogeneous data processing; New learns new heterogeneous patterns, enhancing the system's adaptability; Forget implements selective forgetting, optimizing the storage structure of heterogeneous knowledge.

[0113] Parameter Adaptation Module (PAM): P(θ) = Adapt(θ,L) + γ·Regularize(θ); where: θ is the model parameter set, controlling the core behavior of heterogeneous data processing; L is the loss function, guiding parameter optimization in heterogeneous scenarios; Adapt performs parameter adjustment to improve the processing accuracy of heterogeneous data; Regularize implements parameter regularization to prevent overfitting in heterogeneous scenarios; γ is the regularization strength, ranging from [0.1,0.4], balancing optimization and generalization.

[0114] The model is continuously optimized using the S(X) selection function and the U(M,X) update function. The three evaluation metrics—Quality, Diversity, and Difficulty—are balanced through λ weights (range [0.2, 0.6]). In the parameter adaptation module (PAM), a complete parameter optimization mechanism is established using the Adapt and Regularize optimizers, along with γ regularization strength (range [0.1, 0.4]).

[0115] A complete scene adaptation system was constructed using the Scene Feature Analyzer (SFA). The system fully utilizes the processing results of the preceding claims, particularly its online learning capabilities, and achieves intelligent adaptation to different scenes through the Parameter Adaptation Module (PAM). This module, as the outermost encapsulation of the entire system, realizes a complete closed loop for heterogeneous data processing.

[0116] Scene Feature Analyzer (SFA): S(C) = Extract(C) + α·Domain(C) + β·Complexity(C); where: C represents the application scene features, characterizing the environmental characteristics of heterogeneous data; Extract extracts the basic features of the scene to understand the processing environment of heterogeneous data; Domain analyzes domain features to adapt to different types of heterogeneous scenes; Complexity evaluates the scene complexity and quantifies the processing difficulty of heterogeneous data; α and β are feature weights that dynamically adapt to the needs of different heterogeneous scenes.

[0117] Parameter Adaptation Module (PAM): P(θ,S)=Adapt(θ,S)*W(S)+γ·Default(θ) where: θ is the model parameter set, controlling the core behavior pattern of heterogeneous data processing; S is the scene feature, guiding the environment adaptation of heterogeneous data processing; Adapt realizes dynamic matching between parameters and scenes, improving the accuracy of heterogeneous processing; W constructs a scene weight matrix, quantifying the degree of influence of different scenes on parameters; Default provides default parameter configuration, ensuring the basic performance of heterogeneous processing; γ is the default weight, ranging from [0.1,0.4], balancing adaptability and stability.

[0118] Scene Optimization Controller (SOC): O(P) = Control(P) + λ·Feedback(P) + μ·Update(P); where: P is the adaptation parameter, guiding scene optimization for heterogeneous data processing; Control executes parameter control to maintain stable operation of heterogeneous processing; Feedback collects scene feedback to achieve closed-loop adjustment of heterogeneous optimization; Update updates the optimization strategy to improve the adaptability of heterogeneous scenes; λ and μ are control weights, dynamically adjusting the optimization direction and intensity.

[0119] Intelligent scene adaptation is achieved through the S(C) analysis function and the P(θ,S) adaptation function. Specifically, the three feature extractors—Extract, Domain, and Complexity—capture comprehensive scene features using α and β weights. The system achieves closed-loop control for scene optimization through the Control, Feedback, and Update modules in the Scene Optimization Controller (SOC), along with λ and μ control weights. In particular, the Default(θ) configuration ensures the system's basic performance through γ weights (range [0.1, 0.4]).

[0120] A table-formula region joint attention network (TFJAN module) achieves unified processing of table structures and mathematical formulas. An adaptive feature fusion mechanism is established using the formula TF(D)=∑[α_i·TC_i(D)]*∑[β_j·FC_j(D)]. Channel-wise attention is employed for intelligent interaction of heterogeneous features, overcoming the limitations of traditional separate processing. A multi-level feature extraction and fusion mechanism is implemented, designing a multi-scale symbolic feature extractor (MSFE) and employing a combination strategy of convolutional kernels with different scales {3,5,7}. Dynamic feature balancing is achieved through an adaptive weight system (α_i,β_j), introducing a fusion mechanism of global contextual features and local detail features.

[0121] A deep semantic understanding framework proposes the Formula Semantic Understanding Network (FSUN), which combines BiLSTM and tree structure features to achieve bidirectional semantic modeling and incorporates table context information to enhance the accuracy of formula understanding. An adaptive mechanism with a dynamic weight range [0.4, 0.8] is designed to balance sequence and structural features.

[0122] The adaptive validation and error correction system constructs a multi-level validation network (MLVN) with three layers of validators (syntax, semantics, and logic); it adopts a hybrid error correction strategy that combines historical error correction records and similar case references; and it introduces a dynamic threshold adjustment mechanism to ensure processing quality.

[0123] The scenario adaptive optimization mechanism is designed with a scenario feature analyzer (SFA) to achieve intelligent adaptation to different application scenarios; the parameter adaptive module (PAM) is used to achieve dynamic optimization of model parameters; and a feedback optimization loop is introduced to build a complete scenario optimization closed loop.

[0124] Incremental learning and dynamic optimization are employed, and an Intelligent Sample Filter (ISS) is proposed to automatically select high-quality samples. The Incremental Update Network (IUN) enables continuous optimization of the model. A selective forgetting mechanism is introduced to optimize the knowledge storage structure.

[0125] A structural consistency guarantee mechanism is implemented by designing a structural consistency optimization module (SCO) to ensure the coherence of the overall table structure; accurate structured parsing is achieved through a hierarchical table structure analysis network (HTSN); and an edge attention mechanism is introduced to handle complex spatial relationships.

[0126] Modular integration and optimization, standardized module interfaces are designed to ensure system scalability; intelligent integration of results from various modules is achieved through a multi-source information fusion (MSIF) fusion device; and a cross-validation mechanism is introduced to ensure consistency of processing results.

[0127] Together, they have constructed a complete technical system that significantly improves the processing capabilities of heterogeneous data, possessing significant technical value and application prospects. Each innovation specifically addresses a particular problem in existing technologies, forming a unique technological advantage.

[0128] Significant technical achievements have been made in the field of intelligent parsing and understanding of heterogeneous data. Firstly, regarding recognition accuracy, a Joint Attention Architecture (TFJAN) is employed to unify the processing of tables and formulas, improving recognition accuracy by 25%-30% compared to traditional separate methods. This is mainly due to the fact that the joint attention mechanism overcomes the limitations of the fragmented processing of traditional methods, achieving deep integration of table structure and formula content. Simultaneously, the Multi-Scale Feature Extractor (MSFE) uses a {3,5,7} convolutional kernel combination strategy to ensure the complete capture of symbolic features at different scales, thereby improving the recognition accuracy of complex mathematical symbols to over 95%.

[0129] In terms of semantic understanding capabilities, the deep semantic understanding framework (FSUN) effectively integrates sequence and structural information by combining BiLSTM and tree structure features, achieving a semantic understanding accuracy of over 90% for complex formulas. The introduction of the context-enhanced understanding module (CEUM) further improves the system's ability to understand formula semantics in tabular environments, with an improvement of 40%. In particular, the adaptive mechanism with a dynamic weight range [0.4, 0.8] improves the robustness of semantic parsing by 35%.

[0130] In terms of processing efficiency, the adaptive verification and error correction system significantly improves system performance. The three-layer validator architecture of the Multi-Level Verification Network (MLVN) enables rapid error location and classification, improving system processing efficiency by 45%. The Intelligent Error Correction Mechanism (ICM) accelerates the correction process by incorporating historical experience, reducing the correction time for complex errors by 60% and increasing the error correction accuracy to 92%.

[0131] In terms of scene adaptability, significant breakthroughs have been achieved through the Scene Feature Analyzer (SFA) and Parameter Adaptation Module (PAM). Precise analysis of scene features and dynamic optimization of parameters, combined with a closed-loop feedback mechanism, have reduced the system's adaptation time to different application scenarios by 70%, with performance improvements consistently exceeding 85%. The Scene Optimization Controller (SOC), through a feedback mechanism, further enhances the system's success rate in adapting to new scenarios, reaching 95%.

[0132] Significant progress has also been made in scalability. The combination of the incremental learning framework and the Intelligent Sample Filter (ISS) has improved the system's learning efficiency by 65% ​​when processing new types of data, and achieved an accuracy rate of 88% in selecting effective samples. The standardized modular interface design not only supports flexible system expansion but also improves expansion development efficiency by 75% and reduces maintenance costs by 50%.

[0133] Finally, regarding system stability, the application of Structural Consistency Optimization (SCO) and Multi-Source Information Fusion (MSIF) improved the system's stability in complex environments by 55%, and the consistency of processing results increased to 93%. The introduction of Dynamic Optimizer (DO) reduced the system's performance degradation over long-term operation by 80%. This is due to the structural consistency guarantee and multi-source information fusion ensuring the reliability of processing results, and the dynamic optimization mechanism maintaining the system's long-term stability.

[0134] In summary, this application has achieved significant technical results in multiple aspects, including recognition accuracy, semantic understanding, processing efficiency, scenario adaptability, scalability, and system stability, through breakthroughs in algorithm design, architecture optimization, and mechanism innovation, providing a comprehensive and efficient solution for the field of heterogeneous data processing.

[0135] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problem proposed in this application; however, this does not mean that other units are absent from this embodiment.

[0136] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0138] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A heterogeneous data parsing and understanding system based on artificial intelligence adaptive feature fusion, characterized in that, The system includes: The TFJAN module is used to extract table structure features and mathematical formula region features from heterogeneous data through a multi-level table detector and formula locator; an adaptive weighting mechanism is used to fuse the table structure features and mathematical formula region features to obtain a joint feature representation of tables and formulas. The MSCAM module is used to extract mathematical symbol features from the TFJAN module, process the mathematical symbol features through convolution and pooling layers and graph attention network, construct the spatial dependency relationship of mathematical symbols in heterogeneous environment, perform context awareness on the mathematical symbol features, and enhance the mathematical symbol features. The SCO module is used to extract table hierarchical structure features and formula cell features from the heterogeneous data, and to perform structural consistency processing on the heterogeneous data through structure regularization and structure optimization functions. The FSPA module is used to capture the structured feature representation of mathematical formulas from the MSCAM module, and parse the structured feature representation of mathematical formulas to obtain the content representation of mathematical formulas; The FSUN module is used to extract the content representation of the mathematical formula from the FSPA module and parse the semantic information of the mathematical formula through a deep learning model.

2. The system according to claim 1, characterized in that, The TFJAN module includes: The table feature extraction unit extracts table structural features through a multi-layered convolutional network; The formula feature extraction unit extracts features of the mathematical formula region through a multi-scale feature pyramid. The adaptive weighting unit dynamically optimizes the fusion weights of the table structure features and mathematical formula features through backpropagation.

3. The system according to claim 2, characterized in that, The TFJAN module includes: TF(D)=∑[α_i·TC_i(D)]*∑[β_j·FC_j(D)]; Where: D represents heterogeneous data with a dimension of 1024×1024×3; α_i and β_j are adaptive weights; TC_i is the i-th layer table detector, i∈[1,3], and the output feature dimensions are 256×256×64, 128×128×128, and 64×64×256 respectively, which are extracted through a multi-layer convolutional network; FC_j is the j-th layer formula locator, j∈[1,4], which extracts the regional features of mathematical formulas through the scale feature pyramid; The symbol * represents feature fusion based on channel-wise attention.

4. The system according to claim 3, characterized in that, The MSCAM module includes: M(S)=σ(Conv(Pool(S)))*S+γ·RelationMap(S); Where: S is the symbolic feature tensor obtained from the TFJAN module, batch_size = 32; Conv uses a 3×3 convolution kernel, stacked in three layers, with 64, 128 and 256 channels respectively, to extract discriminative features of symbols in heterogeneous environments layer by layer; Pool output has a fixed dimension of 7×7; γ dynamically adjusts within a range of [0.1, 0.9]. RelationMap is based on graph attention networks with a neighborhood size of K=8, establishing spatial dependencies of symbols in heterogeneous environments.

5. The system according to claim 4, characterized in that, The SCO module includes: S(T)=Optimize(H(T),FC(C))+λ1·Regular(T); Where: H(T) is the table hierarchy feature; FC(C) is the formula cell feature; Regular is a structure regularization term that constrains the structure of heterogeneous tables. λ1 is a balancing factor, ranging from [0.3, 0.7], which dynamically adjusts the weights for structure optimization and normalization; Optimize is the structure optimization function.

6. The system according to claim 5, characterized in that, The FSPA module includes: FS(F)=SoftMax(QF·KF^T / √d)·VF+λ2·StructureGate(F); Among them, QF, KF, and VF are constructed based on the output features of MSCAM, with a dimension of 256, and are used to capture the structured feature representation of mathematical formulas in heterogeneous data; d represents the feature dimension, used for normalization; StructureGate identifies special structural elements and handles complex mathematical expressions in heterogeneous data. λ2 adaptive adjustment range [0.2, 0.8], dynamic equilibrium general characteristics and special structural characteristics.

7. The system according to claim 6, characterized in that, The FSUN module includes: U(P,C)=Parse(P)+α·Context(C)+β·Logic(P,C); Wherein: P is the formula content extracted by the FSPA module, which carries the core semantic information of mathematical formulas in heterogeneous data; C represents table context information, which comes from the TFJAN module and is used to establish a semantic association between formulas and table structures. α and β are dynamic weights, ranging from [0,1], which adaptively adjust semantic parsing and context understanding; Parse, Context, and Logic are implemented using deep neural networks to construct a multi-level semantic understanding framework for heterogeneous data.

8. The system according to any one of claims 1 to 7, characterized in that, The system also includes: A multi-level validation network is used to perform multi-level validation on the structure of the parsed formula to ensure the syntactic, semantic and logical rationality of the formula. An intelligent error correction mechanism is used to correct parsing errors based on historical error correction records and similar case references; An adaptive optimization strategy is used to optimize the parsing process based on an optimization function and dynamic weight adjustment rules.

9. The system according to claim 8, characterized in that, The system also includes: A multi-source information fusion unit is used to fuse table features, structural analysis results, symbol recognition results, semantic parsing results, and verification and error correction results, and to perform deep combination based on dynamic fusion weights; The consistency evaluation module is used to evaluate the accuracy of the fusion results and calculate the similarity between the parsed results and the standard template library; The dynamic optimizer is used to optimize the parsing results and adjust the optimization weights based on temporal features.

10. The system according to claim 9, characterized in that, The system also includes: The intelligent sample filter is used to assess the quality of newly input tabular formula samples and filter them based on sample quality, diversity, and difficulty. Incremental update networks are used to dynamically update the system model, learn new data patterns based on new samples, and simultaneously execute a selective forgetting mechanism to optimize the storage structure. The parameter adaptation module is used to perform parameter adjustments and prevent overfitting through regularization control.

Images