A Multimodal Data Fusion Method Based on Fiscal and Tax Ontology

By adopting a multimodal data fusion method based on fiscal and tax ontology, the semantic confusion and compliance issues in multimodal data fusion were resolved, achieving accurate data mapping and dynamic optimization, and improving data quality and system adaptability.

CN123116142APending Publication Date: 2026-11-17上海立业乐信息科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611295662.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies lack an ontology for the financial and tax domain, leading to semantic confusion during multimodal data fusion, making it difficult to achieve accurate mapping and compliance. Furthermore, the lack of dynamic adjustment capabilities results in high error rates and high maintenance costs.

Method used

A multimodal data fusion method based on the fiscal and tax ontology is adopted, which includes loading a pre-built fiscal and tax domain ontology, performing multimodal data preprocessing, semantic parsing and mapping, and establishing a dynamic feedback optimization closed loop by combining compliance verification and quality verification.

Benefits of technology

It achieves unified fusion of multimodal data at the semantic level, ensuring compliance and accuracy, reducing error rates and maintenance costs, and possessing continuous evolution capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116142A_ABST
    Figure CN123116142A_ABST
Patent Text Reader

Abstract

This invention provides a multimodal data fusion method based on a financial and tax ontology, comprising: loading a pre-constructed financial and tax domain ontology; acquiring and preprocessing multimodal data in financial and tax business scenarios to obtain standardized multimodal data; performing semantic parsing on the standardized multimodal data based on the financial and tax domain ontology to generate a unified semantic representation, and mapping the semantic representation to concepts and attributes in the ontology to establish a mapping relationship; and performing semantic fusion on the multimodal data according to the mapping relationship to generate an ontology-based fusion dataset. This invention ensures the quality of fused data through a dual mechanism of embedded compliance verification and quality verification, and achieves continuous iterative optimization of the fusion strategy through a dynamic feedback closed loop. Furthermore, by using the financial and tax domain ontology as a semantic benchmark to drive the entire fusion process, it solves the semantic disconnect and compliance risks caused by the lack of ontology support in existing technologies, significantly improving the accuracy and business support capabilities of financial and tax multimodal data fusion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a multimodal data fusion method based on the ontology of finance and taxation, which is applied to intelligent financial and taxation business scenarios. Background Technology

[0002] Currently, in the field of financial and tax services, a large amount of multimodal data is generated, including invoice images, accounting voucher texts, financial statement tables, and audio of financial and tax consultations. Effectively integrating this multimodal data, which is scattered from different sources and has different formats, is a key prerequisite for realizing intelligent financial and tax processing.

[0003] However, existing technologies generally lack a core framework capable of unifying and interpreting the semantics of fiscal and tax data—that is, a fiscal and tax domain ontology. Due to this lack of ontology support, when fusing data fields with different modalities and terminology, only format-level splicing can be achieved, failing to understand the underlying fiscal and tax business semantics. This results in semantically chaotic fusion results, making it difficult to support business decisions. This semantic gap directly leads to poor heterogeneous data processing capabilities, a lack of effective semantic alignment and accurate mapping mechanisms, resulting in high fusion error rates and difficulty in guaranteeing data quality.

[0004] Furthermore, due to the lack of an ontology as a mapping benchmark, existing technologies cannot establish a precise matching relationship between multimodal data and financial and tax business processes and compliance rules. This results in a disconnect between the fused data and business applications, failing to directly support specific tasks such as tax declaration and audit verification. In terms of compliance, the existing fusion process is completely independent of financial and tax regulations, lacking real-time verification steps for invoice compliance, accounting standards, and tax laws. This allows non-compliant data to freely enter the fusion results, posing serious financial and tax compliance risks.

[0005] Furthermore, existing integration strategies are typically static and fixed, unable to be dynamically adjusted or customized according to updates to tax and financial regulations, changes in business needs, or the characteristics of enterprise size. This results in high maintenance costs and a high risk of errors. Simultaneously, existing solutions generally lack a feedback optimization loop from integration results to integration strategies. When integrated data is used by subsequent systems and errors are discovered, these errors cannot be automatically captured and used to correct and optimize integration rules, leading to a lack of continuous system evolution capabilities and a failure to gradually improve data quality with use. Summary of the Invention

[0006] This invention is made to solve the above problems, and its purpose is to provide a multimodal data fusion method based on the ontology of finance and taxation.

[0007] This invention provides a multimodal data fusion method based on fiscal and tax ontology, characterized by the following features: S1. Load the pre-built ontology of the finance and taxation domain; S2. Acquire multimodal data in financial and tax business scenarios; S3. Preprocess the multimodal data to obtain standardized multimodal data; S4. Based on the ontology of the finance and taxation domain, perform semantic parsing on standardized multimodal data to generate a unified semantic representation, and map the semantic representation to the concepts and attributes in the ontology of the finance and taxation domain to establish a mapping relationship; S5. Based on the mapping relationship, perform semantic fusion on the multimodal data to generate an ontology-based fusion dataset.

[0008] Further, step S2 includes: S21. Match the corresponding data collection scenario according to the user's business needs, and determine the data types and collection scope to be collected; S22. Collect data for the corresponding modality through multiple channels, including text data, image data, tabular data, and audio data; S23. Perform initial screening on the collected multimodal data to filter out invalid and duplicate data.

[0009] Further, step S3 includes: S31. Clean the text data, segment it into words and semantically annotate it, extract the core semantic information and convert it into a standardized text format; S32. Perform noise reduction, correction, and text recognition on the image data to extract structured text information; S33. Standardize the format of the tabular data and handle outliers to extract structured data; S34. Perform noise reduction and speech-to-text processing on the audio data to extract core semantic information; S35. Verify the integrity and standardization of the preprocessed data to generate a standardized multimodal dataset.

[0010] Furthermore, step S4 generates a unified semantic representation, including: calling the enhanced large language model to perform semantic parsing on the standardized multimodal data, and generating a semantic representation based on the concept-attribute-relation triplet form of the fiscal and tax domain ontology.

[0011] Further, step S5 includes: S51. Load ontology-driven fusion rules, which include semantic fusion rules, data priority rules, and ontology constraint rules. S52. Based on the mapping relationship, align the semantics of data from different modalities to eliminate semantic ambiguity; S53. Adopt a hierarchical fusion strategy, first perform intra-modal fusion of data within the same modality, and then perform cross-modal fusion of data between different modalities, while following ontology constraint rules during the fusion process; S54. Convert the fused data into a structured format that the ontology can recognize, associate it with the corresponding ontology concepts and attributes, and generate an ontology-based fused dataset.

[0012] Furthermore, it also includes: S6, loading tax compliance rules, and performing compliance verification on the semantic fusion process and / or the generated ontology fusion dataset; S7. Mark compliance anomalies found during verification and trigger automatic correction or push them to a manual terminal for processing.

[0013] Furthermore, it also includes: S8, performing multi-dimensional quality verification on the ontology-based fusion dataset, which includes accuracy verification, completeness verification, consistency verification, and timeliness verification; S9. Assess the quality level based on the verification results, and trigger a re-fusion process for unqualified fusion results.

[0014] Furthermore, it also includes: S10, converting the ontology-integrated dataset into a format to adapt to the data requirements of external financial and tax intelligent agents, workflows, or tool calling modules, and pushing it to external modules; S11. Receive data usage or data anomaly information from external modules and generate feedback data.

[0015] Furthermore, it also includes: S12, collecting feedback data, which includes feedback from quality verification, feedback from external modules, user feedback, and updated information on financial and tax regulations; S13. Analyze the feedback data and generate optimization instructions for the fusion strategy or mapping rules; S14. According to the optimization instructions, update the fusion strategy, ontology mapping rules or preprocessing parameters, and apply the updated strategy to the subsequent multimodal data fusion process to form a dynamic iterative optimization closed loop.

[0016] The role and effect of invention The present invention provides a multimodal data fusion method based on a fiscal and tax ontology. By loading a pre-built fiscal and tax domain ontology, this ontology serves as the core framework for governing and interpreting the semantics of fiscal and tax data, driving the entire fusion process of multimodal data. This solves the semantic gap problem caused by the lack of ontology support in existing technologies, enabling the fusion results to be unified at the semantic level and directly supporting upper-level fiscal and tax business decisions.

[0017] Meanwhile, this invention ensures that the merged data meets business usability standards in both compliance and accuracy by embedding a dual verification mechanism of financial and tax compliance verification and quality verification during the fusion process, effectively reducing financial and tax compliance risks. By establishing a dynamic feedback optimization closed loop from fusion results to fusion strategies, this method can continuously and automatically capture error information and optimization needs from downstream business systems, and iteratively update fusion strategies and mapping rules accordingly. This enables the system to continuously evolve and improve data quality as it is used, significantly reducing maintenance costs. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a multimodal data fusion method based on a financial and tax ontology, provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following embodiments are described in detail with reference to the accompanying drawings.

[0020] like Figure 1 As shown, this invention provides a multimodal data fusion method based on financial and tax ontology. This method can be applied to scenarios such as intelligent enterprise financial and tax management systems, bookkeeping software service platforms, and financial and tax large-scale model knowledge base governance systems. The following section uses the application of this method in a financial and tax automation system as an example to explain each execution step in detail.

[0021] Step S1: Load the pre-built tax and finance domain ontology.

[0022] During system startup or initialization, the program loads a pre-built financial and tax domain ontology. This ontology is organized using a three-layer architecture consisting of a core layer, a business layer, and a rules layer.

[0023] The core layer defines the fundamental concepts and attributes in the field of finance and taxation. For example, the concept of "invoice" includes attributes such as invoice code, invoice number, invoice date, amount, tax amount, total price including tax, seller's name, seller's tax ID, buyer's name, buyer's tax ID, tax rate, and invoice type. The data types for amount and tax amount are monetary values, while the data type for invoice type is an enumeration. The concept of "taxpayer" includes attributes such as taxpayer identification number, taxpayer name, taxpayer type, and industry. The concept of "accounting voucher" includes attributes such as voucher number, voucher date, summary, debit account, credit account, and amount. The concept of "financial statement" includes attributes such as statement type, reporting period, total assets, total liabilities, operating revenue, and total profit.

[0024] The business layer defines the relationships between various concepts in the financial and tax business scenario. For example, "invoice" and "taxpayer" are associated through the "issued by..." relationship, indicating that the invoice is issued by the seller taxpayer; "taxpayer" and "invoice" are associated through the "receive" relationship, indicating that the buyer taxpayer receives the invoice; "accounting voucher" and "invoice" are associated through the "entry basis" relationship, indicating that the accounting voucher is based on the invoice; "tax return" and "financial statement" are associated through the "declaration basis" relationship, indicating that the tax declaration is based on the financial statement. These relationships are stored in the form of directed edges, allowing for the rapid retrieval of the complete business chain associated with a concept using graph traversal algorithms (such as breadth-first search or depth-first search). For example, starting from an "invoice" node, following the "entry basis" edge leads to the "accounting voucher" node, and following the "summarize to" edge further leads to the "financial statement" node, thus completely reconstructing the entry and report aggregation path of the invoice.

[0025] The rules layer defines compliance constraints and business rules in machine-executable form, encoded using structured "IF-THEN" expressions. For example, the compliance rule "Invoice amount must be greater than zero" is encoded as: IF invoice.amount <= 0 THEN flag an exception ("Invoice amount is invalid") AND prevent merging. The compliance rule "Buyer's tax ID must be 15 or 18 digits" is encoded as: IFLEN(invoice.buyer's tax ID) NOTIN(15, 18) THEN flag an exception ("Buyer's tax ID format is incorrect") AND prevent merging. The compliance rule "Invoice tax rate must be a statutory tax rate" is encoded as: IF invoice.tax rate NOTIN(0.06, 0.09, 0.13) THEN flag an exception ("Invoice tax rate is non-compliant"). The consistency rule "The difference between the amount extracted from the invoice image OCR and the amount extracted from the table shall not exceed 0.01 yuan" is encoded as: IF ABS(invoice.image_OCR_amount - invoice.table_amount) > 0.01 THEN mark an exception ("Multimodal amount inconsistency") AND abort fusion. The business rule "The amount of the accounting voucher must be equal to the total price and tax of the associated invoice" is encoded as: IF accounting voucher.amount != SUM(associated invoice.total price and tax) THEN mark an exception ("Voucher amount does not match invoice"). These rules are serialized into JSON format and mounted on the corresponding concept nodes, allowing the validation engine to directly load and execute the rules by parsing the JSON.

[0026] The ontology is stored using a graph database. Concepts are stored as graph nodes, each containing a concept name and a list of attributes; relations are stored as directed edges, each containing a relation name and relation type; rules are stored as additional attributes of nodes. The system uses a graph database query language to retrieve and traverse ontology concepts. In addition, the system loads user requirement configuration data (including data collection scenario preferences and fusion strategy preferences), a tax and financial regulations database, an expert rule base, an enhanced large language model invocation configuration, and communication interface configurations with preceding external modules (preferably tax and financial intelligent agents, workflow engines, or tool invocation modules) to complete system initialization.

[0027] Step S2: Obtain multimodal data in the context of financial and tax business scenarios.

[0028] The program matches the corresponding data collection scenario based on the user's business needs, determining the data type and scope to be collected. Data collection scenario matching supports two implementation methods. The first method involves the user manually selecting from preset scenario templates on the front-end interface, such as "Accounting Agency Data Collection," "Tax Declaration Data Collection," or "Audit Data Collection." Each template predefines the required data type and scope. The second method involves the user inputting their business needs in natural language. The program then calls an enhanced large language model for semantic understanding, parsing the natural language requirements into structured collection parameters and automatically matching the corresponding data collection scenario template.

[0029] Once the data collection scenario is determined, the program collects data for the corresponding modalities through multiple channels. Text data (including accounting vouchers, tax returns, and contract texts) is obtained through API interfaces connecting to the enterprise's ERP or financial software systems, or imported via file upload. Image data (including invoice images, voucher photos, and scanned reports) is obtained through scanning device interfaces, uploaded via mobile device photography, or automatically captured from email attachments. Tabular data (including financial statements and tax returns) is imported via Excel files, or structured data is extracted from the financial system via API interfaces. Audio data (including recordings of financial and tax consultations and meeting recordings) is recorded in real time through audio acquisition devices, or imported via file upload of historical recordings.

[0030] After data collection, the program performs initial screening of the multimodal data. For image data, sharpness detection is performed, filtering out blurry images with sharpness scores below a preset threshold; for text data, null value detection is performed, filtering out blank text; for audio data, signal-to-noise ratio detection is performed, filtering out audio with excessive noise. Simultaneously, hash value comparison is performed on invoice image and text data to detect and remove duplicate data; uniqueness verification is performed on combinations of invoice numbers and invoice codes, retaining only one copy of any combination appearing multiple times. Abnormal data, such as incorrectly formatted table files or invoice data missing key fields, is marked and the anomaly type is recorded.

[0031] Step S3: Preprocess the multimodal data to obtain standardized multimodal data.

[0032] The program performs targeted preprocessing on data of different modalities. For text data, redundant characters and special symbols are removed, and a word segmentation dictionary specifically for the financial and tax field is used for word segmentation. An enhanced large language model is then called for entity recognition and semantic annotation to extract core semantic information such as invoice number, amount, taxpayer identification number, and accounting subject, and finally convert it into standardized JSON format text.

[0033] For image data, Gaussian filtering is used for noise reduction, Hough transform is used to detect straight lines and calculate tilt angles for rotation correction, document edges are detected and cropped to the effective area, and then text information (including invoice number, invoice date, amount, etc.) in the image is extracted by an OCR recognition engine (such as Paddle OCR), converted into structured text data, and stored in association with the original image data.

[0034] For tabular data, standardize the format, unify the table header naming and data type (e.g., unify the date format to YYYY-MM-DD), fill missing numerical fields with the average of similar data or the previous period's data and mark them as estimated values, detect and locate anomalies such as imbalance between "Total Assets" and "Total Liabilities and Owner's Equity" in the balance sheet, and extract the core data from the table and convert it into structured data.

[0035] For audio data, spectral subtraction is used to remove environmental noise, speech activity detection is used to remove silent and non-speech segments, and a speech recognition engine (such as Whisper) is used to convert the audio into text data, extract core semantic information such as consultation questions and solutions, and store them in association with the original audio data.

[0036] Finally, the program performs integrity checks (checking if all required fields are present) and standardization checks (checking if the format conforms to specifications) on the preprocessed multimodal data. Once the checks pass, a standardized multimodal dataset is generated. Each record in this dataset contains a unified data identifier, modality type, original data reference path, preprocessed structured extraction information, and preprocessing status markers.

[0037] Step S4: Based on the ontology of the finance and taxation domain, perform semantic parsing, mapping, and establish mapping relationships.

[0038] The program first extracts the ontology rule set corresponding to the current business scenario from the financial and tax domain ontology. The extraction method starts with the core concepts associated with the business scenario and uses graph traversal queries in a graph database to obtain related concept nodes and their attribute definitions and rule constraints. For example, the core concepts associated with the "agency accounting" scenario are "invoice," "accounting voucher," "accounting subject," and "taxpayer." The program extracts all attribute definitions and rule constraints for these concepts to generate an ontology mapping rule set.

[0039] Subsequently, the program invokes the enhanced large language model to perform semantic parsing on the standardized multimodal data, generating a unified semantic representation. The enhanced large language model is obtained by domain adaptation of the general large language model through the Prompt project and / or instruction fine-tuning methods. The Prompt project enhancement involves automatically constructing a structured Prompt containing tax ontology concept definitions, attribute lists, and a small number of labeled examples when invoking the large language model, guiding the model to parse according to a predefined output format. The instruction fine-tuning enhancement involves using a tax domain labeled dataset to efficiently fine-tune the parameters of the base large language model (e.g., LoRA tuning), enabling the model to more accurately understand tax terminology and context. The unified semantic representation output by the large language model adopts a concept-attribute-relation triplet form based on the tax domain ontology. Each triplet contains a subject (data instance identifier), an attribute (full path name of the ontology attribute), and a value (extracted attribute value).

[0040] The program matches the list of triples generated by the large language model with the concepts and attributes of the tax domain ontology, establishing a mapping relationship between each modal of data and the ontology. The matching method involves an exact match between the "attribute" field of the triple and the complete path of the ontology concept attribute (formatted as "concept name.attribute name"). When mapping conflicts occur (such as the same ontology attribute being mapped by multiple data sources with inconsistent values), the program invokes expert rules and the large language model to collaboratively resolve the conflict. For conflicts with explicit rules, expert rules are prioritized (e.g., for numerical fields like tax amounts, tabular data sources have higher confidence than text data sources, so tabular values ​​are preferred). For complex semantic conflicts that expert rules cannot cover, the large language model is invoked to perform contextual semantic judgment to determine the optimal mapping relationship. Finally, consistency checks (checking whether the mapping relationship is consistent with the concept hierarchy and attribute constraints defined in the ontology) and accuracy checks (extracting a portion of the mapping results for semantic consistency verification by the large language model) are performed on the mapping relationships, generating the final multimodal data-ontology mapping list.

[0041] Step S5: Perform semantic fusion based on the mapping relationship to generate an ontology-based fusion dataset.

[0042] The program loads ontology-driven fusion rules, including semantic fusion rules (defining the merging manner of multi-modal data under the same ontology concept), data priority rules (defining the credibility priority of data from different modalities and different sources), and ontology constraint rules loaded from the ontology rule layer.

[0043] Based on the mapping relationships in the mapping list, the program aligns the semantics of different modal data to eliminate semantic ambiguity. For example, the amount identified by OCR in invoice image data is "1000.00", the converted uppercase amount "One Thousand Yuan Only" in text data is also "1000.00", and the amount in table data is "1000.00". The three are consistent in value, confirmed to be semantically consistent and unified as the value "1000.00". In case of inconsistency (such as deviation caused by OCR recognition error), data from the high-credibility source is adopted according to the data priority rule, and a semantic alignment log is recorded.

[0044] Next, the program adopts a hierarchical fusion strategy. The first layer is intra-modal fusion, which performs internal fusion on multiple data instances of the same type of modality (for example, for multiple photos of the same invoice taken from different angles, the OCR results of each copy are first merged to complement missing information, and conflicting fields are selected according to confidence). The second layer is cross-modal fusion, which performs cross-modal fusion on data of different types of modalities (for example, fusing the semantic representation of invoice image data with the semantic representation of voucher text data), associates data instances of different modalities through relationships in the ontology (such as the "accounting basis" relationship), and follows ontology constraint rules during the fusion process (such as the rule "the amount of the voucher must be equal to the total price and tax of the associated invoice").

[0045] After the fusion is completed, the program converts the fused data into a structured format recognizable by the ontology, associates the corresponding ontology concepts and attributes, and retains the association relationships of each original modal data to ensure data traceability, so as to generate an ontological fused dataset.

[0046] Steps S6 and S7: Compliance verification and exception handling.

[0047] The program loads fiscal and tax compliance rules associated with the current fused data from the fiscal and tax regulation database and the rule layer of the ontology. The rules are stored in a structured form, and each rule includes a rule ID, a rule type, applicable concepts, trigger conditions and execution actions.

[0048] The program performs compliance checks on both the semantic fusion process and the generated ontology-based fusion dataset. Process compliance checks include verifying whether the alignment result triggers compliance rules during the semantic alignment stage (e.g., if a non-statutory tax rate value of 0.05 is found during alignment, an anomaly is flagged and the data is prevented from participating in subsequent fusion), and verifying whether the fusion operation triggers consistency rules during the cross-modal fusion stage (e.g., when merging invoice image data and accounting voucher text data, if the difference in amount exceeds 0.01 yuan, the fusion of that record is stopped and an anomaly is reported). Result compliance checks involve iterating through all records in the fusion dataset and executing all compliance rules bound to each ontology concept associated with that record (e.g., rules such as invoice amount must be greater than zero, invoice tax rate must be a statutory tax rate, and accounting voucher amount must equal the sum of the invoice price and tax, etc.).

[0049] For compliance anomalies detected during verification, the program first marks the abnormal data, recording the anomaly type, anomaly fields, and triggering rules. Then, it attempts automatic correction, calling correction rules from the expert rule base and a large language model to determine if the anomaly can be automatically corrected (for example, if the difference between an invoice amount "1000" and a voucher amount "1000.00" is only a decimal formatting issue, it will be automatically corrected; if the invoice tax number is missing a digit, the large language model will be used to infer the complete tax number based on the context, and it will be automatically completed if the confidence level is higher than a preset threshold). For anomalies that cannot be automatically corrected, a structured anomaly report containing the anomaly type, triggering rules, anomaly data details, and correction suggestions is generated and pushed to the human terminal for operators to correct. After correction, the process re-enters the fusion and verification process.

[0050] Steps S8 and S9: Quality verification and grade assessment.

[0051] The program performs multi-dimensional quality checks on the ontology-based fusion dataset that has passed compliance verification. Accuracy verification is performed by comparing the fusion results with the original multimodal data and ontology rules. Integrity verification checks whether the fusion results are missing any key data fields necessary for the business. Consistency verification checks the semantic consistency of the fused data from different modalities, such as whether there are contradictions in different related records for the same business. Timeliness verification checks whether the fused data meets business timeliness requirements (e.g., in an agency accounting scenario, invoices for the current month must be collected and fused before the 15th of the following month).

[0052] Based on the verification results, the program assesses the quality level of the fusion results. Results that pass all four dimensions are rated as excellent and can proceed directly to the next stage. Results that pass accuracy and completeness but have minor issues with consistency or timeliness (excluding critical fields) are rated as acceptable and can proceed to the next stage, but with an attached quality flag. Results that fail accuracy or completeness are rated as unacceptable, triggering a re-fusion process. The process reverts to the semantic alignment step, adjusts parameters, or changes the data source before re-fusion.

[0053] Steps S10 and S11: Data exchange and feedback reception with external modules.

[0054] The program converts the merged data to a suitable format to meet the data requirements of external financial and tax intelligence agents, workflow engines, or tool calling modules. The conversion method involves pre-defining data format adapters for each module. Each adapter includes field mapping rules and data type conversion rules. The program selects the appropriate adapter to perform the conversion based on the target module.

[0055] After the transformation is complete, the program proactively pushes the adapted fused data to external modules via message queues or RESTful API interfaces, supporting collaborative decision-making by intelligent agents, node execution of the workflow engine, and intelligent invocation of tool calling modules. Simultaneously, the program subscribes to events in the message queue to receive feedback from external modules on data usage (including the frequency and hit rate of fused data usage) and data anomaly information (including data conflicts, errors, and missing data discovered by intelligent agents during use), organizing this information into structured feedback data.

[0056] Steps S12 to S14: Dynamic feedback optimization and fusion strategy iteration.

[0057] The program collects feedback data from multiple sources, including quality verification feedback (quality level assessment results, details of non-conformance records) generated in steps S8 and S9, feedback from external modules received in step S11, user feedback submitted by users during the personalized integration confirmation and compliance anomaly correction process, and updated information on financial and tax regulations.

[0058] The program invokes the enhanced large language model to perform structured analysis on the feedback dataset and generate optimization instructions. The analysis process includes: classifying the feedback data by problem type (mapping error, compliance omission, quality defect, inappropriate strategy); correlating the fusion record ID in the feedback data to trace back the corresponding fusion log to locate the specific steps and rules that caused the error; and then selecting or combining optimization schemes from a predefined optimization action space. The generated optimization instructions are in a structured format, including the problem type, root cause analysis results, specific optimization actions (such as adjusting the data source priority weight of a field, modifying a mapping conflict resolution rule, updating a preprocessing parameter), and the scope of application.

[0059] The program automatically updates the fusion strategy, ontology mapping rules, and preprocessing parameters based on optimization instructions, and persistently stores the optimized strategy with a version number. During subsequent multimodal data fusion, the system automatically applies the latest version of the strategy. By continuously collecting newly generated feedback data and repeating the above analysis, optimization, and application steps, a dynamic iterative closed loop of "collection-analysis-optimization-application-re-collection" is formed, ensuring that the fusion method always adapts to the latest business needs and regulatory changes. Example

[0060] The following example, using a typical application scenario of an accounting outsourcing software service platform, will further illustrate the technical solution of this invention.

[0061] A certain accounting agency platform provides accounting and tax filing services to many small and medium-sized enterprises. One of its clients is a trading and retail company. During the monthly VAT filing cycle, the platform receives the following multimodal business data from this company: a scanned image of a VAT invoice, text data of accounting vouchers exported from the company's accounting software, and an Excel spreadsheet of the monthly VAT return.

[0062] The system first executes step S1, loading the pre-built financial and tax domain ontology, user requirement configuration data, financial and tax law database, expert rule base, and large language model call configuration. Based on the user requirement configuration data, the enterprise's business scenario is "agency bookkeeping," and the fusion strategy preference is "invoice image data has higher priority than bookkeeping text data."

[0063] Next, step S2 is executed. Based on the "agency accounting" scenario template, the system determines that the data types to be collected are invoice images, accounting voucher text, and declaration forms. After obtaining data from various channels, the invoice images are checked for clarity (clarity score 0.95, passed), and the text and forms are checked for null values ​​(passed). No duplicate data was found.

[0064] Then, step S3 is executed, where the system preprocesses the data for each modality. After denoising and correcting the invoice image, OCR recognition extracts information such as the invoice number "12345678", invoice date "2024-01-15", amount "10000.00", tax amount "1300.00", tax rate "0.13", seller's tax number "91370100MA3XXXXXX", and buyer's tax number "91370200XXXXXXXXXX". The accounting voucher text is cleaned and semantically annotated, extracting information such as the voucher number "JZ-2024-0115-001", voucher date "2024-01-15", debit account "inventory", credit account "bank deposit", and amount "11300.00". The VAT return form was standardized in format, and information such as the current period's output VAT "1300.00", current period's input VAT "800.00", and current period's VAT payable "500.00" was extracted. After verification, a standardized multimodal dataset was generated.

[0065] Then, step S4 is executed. The system extracts the attribute definitions and rule constraints of the concepts (invoices, accounting vouchers, taxpayers, accounting subjects, and tax returns) corresponding to the "agency accounting" scenario from the ontology. The enhanced large language model is invoked to perform semantic parsing on the standardized multimodal data. The large language model embeds the financial and tax ontology concept definitions in the Prompt, outputting a list of triples, such as: {"Subject": "IMG_INV_001", "Attribute": "Invoice.Amount", "Value": "10000.00"}, {"Subject": "IMG_INV_001", "Attribute": "Invoice.Tax Rate", "Value": "0.13"}, {"Subject": "TXT_VOUCHER_001", "Attribute": "Accounting Voucher.Amount", "Value": "11300.00"}, etc. The triples are matched with the ontology concept attributes to establish mapping relationships. No mapping conflicts are found after conflict detection, and a mapping list is generated after consistency verification.

[0066] Next, step S5 is executed. After loading the fusion rules, semantic alignment is performed on the multimodal data. The sum of the invoice image OCR amount "10000.00" and the tax amount "1300.00", "11300.00", is completely consistent with the amount "11300.00" in the accounting voucher text, and the semantic alignment passes. Layered fusion is then performed: first, intramodal processing is performed on the invoice image and the accounting voucher text (without conflicts within the same modality), and then cross-modal fusion is performed, associating the invoice instance and the accounting voucher instance through the "entry basis" relationship in the ontology. During the fusion process, rule R006 "the amount of the accounting voucher must be equal to the total price and tax of the associated invoice" is triggered, and the verification passes. After fusion, an ontology-based fusion dataset is generated, in which the invoice records contain complete invoice attributes and associated accounting voucher references.

[0067] Steps S6 and S7 are then executed. The system loads compliance rules and verifies the fusion result. The invoice amount "10000.00" is greater than zero, the buyer's tax ID is 18 digits, and the tax rate "0.13" is the statutory tax rate; all are compliant. No compliance anomalies were triggered.

[0068] Then, steps S8 and S9 are executed. The fusion results undergo multi-dimensional quality checks: accuracy check compares the fused values ​​with the original data; completeness check confirms that all required fields on invoices and vouchers are present; consistency check finds no contradictions; and timeliness check confirms that the data is within the processing timeframe for the current month. If all four dimensions pass, the quality level is rated as excellent.

[0069] Following steps S10 and S11, the system converts the ontology-based fusion dataset into a data format recognizable by the tax intelligence agent and workflow engine, and then pushes it. Upon receiving the fusion data, the tax intelligence agent combines it with the tax return data to make a VAT declaration decision, calculating the VAT payable as "1300.00 - 800.00 = 500.00", which matches the value entered in the tax return, and no anomalies are reported.

[0070] Finally, steps S12 to S14 are executed. Since no compliance anomalies, quality deficiencies, or external feedback errors occurred during this round of integration, the system records the logs and execution parameters of this integration process as baseline data. In subsequent operations, if tax regulations are updated (such as changes to the statutory tax rate list), the system will automatically capture the update information, generate optimization instructions to update the tax rate enumeration values ​​in the compliance rule base, and apply the updated rules to subsequent integration processes, forming a closed loop of continuous optimization.

[0071] The role and effect of the embodiments This invention presents a multimodal data fusion method based on a tax and finance ontology. By loading a tax and finance domain ontology as a semantic benchmark, it drives the complete process of fusion from collection, preprocessing, semantic mapping, and hierarchical integration of three heterogeneous modal data: invoice images, accounting voucher text, and tax return forms. This achieves precise alignment and unification of cross-modal data at the semantic level of tax and finance business. The embedded compliance verification mechanism automatically checks key fields such as invoice amount, tax rate, and tax number format during the fusion process. The quality verification mechanism evaluates the fusion results from four dimensions: accuracy, completeness, consistency, and timeliness. Both mechanisms jointly ensure the compliance and business usability of the fused data. By pushing the fusion results to an external tax and finance intelligent agent and receiving feedback, it is verified that the fused data can directly support downstream intelligent business decisions, forming a complete closed loop from data fusion to business application and strategy iteration optimization.

[0072] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal data fusion method based on fiscal and tax ontology, characterized in that, include: S1. Load the pre-built ontology of the finance and taxation domain; S2. Acquire multimodal data in financial and tax business scenarios; S3. Preprocess the multimodal data to obtain standardized multimodal data; S4. Based on the tax and finance domain ontology, perform semantic parsing on the standardized multimodal data to generate a unified semantic representation, and map the semantic representation to the concepts and attributes in the tax and finance domain ontology to establish a mapping relationship; S5. Based on the mapping relationship, perform semantic fusion on the multimodal data to generate an ontology-based fusion dataset.

2. The method according to claim 1, characterized in that, Step S2 includes: S21. Match the corresponding data collection scenario according to the user's business needs, and determine the data types and collection scope to be collected; S22. Collect data for the corresponding modality through multiple channels, including text data, image data, tabular data, and audio data; S23. Perform initial screening on the collected multimodal data to filter out invalid and duplicate data.

3. The method according to claim 1, characterized in that, Step S3 includes: S31. Clean the text data, segment it into words and semantically annotate it, extract the core semantic information and convert it into a standardized text format; S32. Perform noise reduction, correction, and text recognition on the image data to extract structured text information; S33. Standardize the format of the tabular data and handle outliers to extract structured data; S34. Perform noise reduction and speech-to-text processing on the audio data to extract core semantic information; S35. Verify the integrity and standardization of the preprocessed data to generate the standardized multimodal dataset.

4. The method according to claim 1, characterized in that, The generation of a unified semantic representation in step S4 includes: The enhanced large language model is invoked to perform semantic parsing on the standardized multimodal data, generating a semantic representation based on the concept-attribute-relation triplet form of the tax domain ontology.

5. The method according to claim 1, characterized in that, Step S5 includes: S51. Load the ontology-driven fusion rules, which include semantic fusion rules, data priority rules, and ontology constraint rules; S52. Based on the mapping relationship, align the semantics of different modal data to eliminate semantic ambiguity; S53. A hierarchical fusion strategy is adopted, first performing intra-modal fusion of data within the same modality, and then performing cross-modal fusion of data between different modalities, while following the ontology constraint rules during the fusion process; S54. Convert the fused data into an ontology-recognizable structured format, associate it with the corresponding ontology concepts and attributes, and generate the ontology-based fused dataset.

6. The method according to claim 1, characterized in that, Also includes: S6. Load the financial and tax compliance rules and perform compliance verification on the semantic fusion process and / or the generated ontology fusion dataset; S7. Mark compliance anomalies found during verification and trigger automatic correction or push them to a manual terminal for processing.

7. The method according to claim 1, characterized in that, Also includes: S8. Perform multi-dimensional quality verification on the ontology-based fusion dataset. The multi-dimensional quality verification includes accuracy verification, completeness verification, consistency verification, and timeliness verification. S9. Assess the quality level based on the verification results, and trigger a re-fusion process for unqualified fusion results.

8. The method according to claim 1, characterized in that, Also includes: S10. Convert the ontology-based fusion dataset to adapt to the data requirements of external financial and tax intelligent agents, workflows, or tool calling modules, and push it to the external modules. S11. Receive data usage or data anomaly information from the external module and generate feedback data.

9. The method according to claim 8, characterized in that, Also includes: S12. Collect feedback data, including feedback from the quality verification, feedback from the external module, user feedback, and updated information on financial and tax regulations. S13. Analyze the feedback data and generate optimization instructions for the fusion strategy or mapping rules; S14. According to the optimization instructions, update the fusion strategy, ontology mapping rules or preprocessing parameters, and apply the updated strategy to the subsequent multimodal data fusion process to form a dynamic iterative optimization closed loop.