Methods and systems for processing multi-source heterogeneous funding data

Abstract

This invention relates to the field of financial data processing technology, specifically a method and system for processing multi-source heterogeneous financial data. The method includes: establishing a data source registry to record source information and uniquely identify the data; performing differentiated parsing on financial data of different formats to generate standardized columnar storage data; traversing the columnar storage data to extract statistical features, classifying fields through metadata analysis and financial dictionary matching, and combining a localized large language model to parse the business connotations of difficult-to-classify fields and map them to a knowledge graph in the table header; based on the field classification and semantic recognition results, applying a rule engine template to generate mapping rules from source fields to an enterprise-level data model; transforming the data structure according to the mapping rules and performing standardization processing; further optimizing quality through data cleaning; and finally verifying the data to output standardized financial data that supports data traceability and meets the stringent requirements of financial regulatory applications.

Description

Technical Field

[0001] This application relates to the field of financial data processing technology, specifically to methods and systems for processing multi-source heterogeneous financial data. Background Technology

[0002] With the deepening of economic globalization and the rapid development of financial technology, the scale and complexity of capital flows have increased significantly. The widespread adoption of new financial instruments and transaction methods such as cross-border transactions, electronic payments, and virtual currencies has made capital flows more convenient, but it has also provided more opportunities and covert means for illegal activities such as money laundering, terrorist financing, corruption, and fraud. Currently, financial data is characterized by its wide range of sources, diverse formats, and high dynamism. Faced with such massive amounts of various types of financial data, traditional methods of monitoring and analyzing funds rely on manual operation, which is inefficient and prone to missing crucial information. Therefore, there is an urgent need for a method that can quickly and accurately process massive amounts of various types of financial data to improve the efficiency of fund monitoring. Summary of the Invention

[0003] This application provides a method and system for processing multi-source heterogeneous financial data, aiming to solve the problems of low efficiency and easy omission of key information in existing technologies when dealing with massive amounts of various types of financial data.

[0004] The first aspect disclosed in this application provides a method for processing multi-source heterogeneous funding data, the method comprising:

[0005] After accessing multi-source heterogeneous funding data, a data source registry is established to record at least the unique identifier of the institution to which each data source belongs, the update frequency, the timeliness mark, and the connection method. Each piece of funding data is also marked with the unique identifier of its corresponding data source, thus establishing the basic architecture of data lineage.

[0006] Based on multi-source heterogeneous funding data carrying data source identifiers, the data format is identified, differential parsing is performed, and standardized columnar storage data is output. The standardized columnar storage data adopts a columnar organization structure, with each column corresponding to a field. It stores the original data value set of the field in all multi-source heterogeneous funding data, forming source field data, and is associated with field metadata. Field metadata is standardized descriptive information that characterizes the essential attributes and business characteristics of source field data.

[0007] The system iterates through the columnar data and its associated field metadata, extracts statistical features, and generates a data feature description file containing field statistical features and metadata information. Based on this file, it performs field semantic recognition and classification.

[0008] By analyzing field metadata and matching it with a financial domain dictionary, fields with significant statistical characteristics and similar business attributes are grouped into the same category. For the remaining unclassified fields, a locally deployed, financial domain-optimized second LLM model is used to parse the true business meaning of the field metadata. The parsing results are then mapped onto a dynamic header-fused knowledge graph to obtain the association between unclassified fields and business entities in the graph.

[0009] Based on the above field classification results and semantic recognition results, a set of mapping relationships from source field data to enterprise-level standardized data models is generated by combining the rule engine template. The rule engine template includes field merging rules, field splitting rules, and field calculation rules.

[0010] Based on the mapping relationship set, the source field data of multi-source heterogeneous financial data is transformed into an enterprise-level standardized data model and standardized processing is performed, including data preprocessing to improve data purity, converting financial and digital data into digital format, and standardizing the offsetting identifier, payment direction and currency type.

[0011] The standardized financial data will be cleaned to further optimize data quality;

[0012] The optimized funding data is then verified for temporal continuity, data integrity, and data continuity to obtain high-quality, traceable, standardized funding data.

[0013] The second aspect disclosed in this application provides a system for processing multi-source heterogeneous funding data, the system comprising:

[0014] The access fund data module is used to access multi-source heterogeneous fund data, establish a data source registry, record at least the unique identifier of the institution to which each data source belongs, update frequency, timeliness mark and connection method, and mark the unique identifier of its corresponding data source for each piece of fund data, thus establishing the basic architecture of data lineage.

[0015] The funds data parsing module is used to identify the data format of multi-source heterogeneous funds data carrying data source identifiers, perform differentiated parsing, and output standardized columnar storage data. The standardized columnar storage data adopts a columnar organization structure, with each column corresponding to a field. It stores the original data value set of the field in all multi-source heterogeneous funds data, forming source field data, and is associated with field metadata. Field metadata is standardized descriptive information that characterizes the essential attributes and business characteristics of source field data.

[0016] The funding data fusion module is used to traverse columnar storage data and its related field metadata, extract statistical features, and generate a data feature description file containing field statistical features and metadata information. Based on this file, field semantic recognition and classification are performed.

[0017] By analyzing field metadata and matching it with a financial domain dictionary, fields with significant statistical characteristics and similar business attributes are grouped into the same category. For the remaining unclassified fields, a locally deployed, financial domain-optimized second LLM model is used to parse the true business meaning of the field metadata. The parsing results are then mapped onto a dynamic header-fused knowledge graph to obtain the association between unclassified fields and business entities in the graph.

[0018] Based on the above field classification results and semantic recognition results, a set of mapping relationships from source field data to enterprise-level standardized data models is generated by combining the rule engine template. The rule engine template includes field merging rules, field splitting rules, and field calculation rules.

[0019] The Funds Data Standardization Processing Module is used to transform the source field data of multi-source heterogeneous funds data into an enterprise-level standardized data model based on a set of mapping relationships, and to perform standardization processing, including data preprocessing to improve data purity, converting funds and digital data into digital formats, and standardizing offsetting identifiers, payment directions, and currency types.

[0020] The funds data optimization module is used to clean the standardized funds data and further optimize the data quality.

[0021] The funds data verification module verifies the time sequence continuity, data integrity, and data continuity of the further optimized funds data to obtain high-quality, traceable, and standardized funds data.

[0022] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0023] 1) High degree of automation, capable of handling heterogeneous data sources of various formats without manual pre-configuration, significantly reducing system deployment and maintenance costs;

[0024] 2) High level of intelligence: Utilizing the semantic understanding capabilities of large language models, it can adaptively access and process new data sources;

[0025] 3) Comprehensive data quality assurance: Establish a multi-level, full-process data quality inspection and monitoring system to ensure the accuracy and reliability of output data;

[0026] 4) Strong traceability: Establish a complete data traceability mechanism to support full-process backtracking and problem localization in data processing;

[0027] 5) High processing efficiency: Through intelligent data parsing, cleaning and verification algorithms, the processing efficiency and accuracy of multi-source heterogeneous financial data are greatly improved.

[0028] In summary, the multi-source heterogeneous funding data processing method provided by this invention automatically and uniformly processes heterogeneous funding data from different platforms into high-quality, standardized data, establishes a complete data quality assurance system, and provides a reliable data foundation for subsequent financial supervision. Attached Figure Description

[0029] Figure 1 This application provides a flowchart illustrating the processing method for multi-source heterogeneous funding data;

[0030] Figure 2 This application provides a schematic diagram of the system structure for processing multi-source heterogeneous funding data.

[0031] Explanation of reference numerals in the attached diagram: 11. Funds data access module; 12. Funds data parsing module; 13. Funds data fusion module; 14. Funds data standardization processing module; 15. Funds data optimization module; 16. Funds data verification module. Detailed Implementation

[0032] This application provides a method and system for processing multi-source heterogeneous funding data. The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. It should be understood that this application is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. It should also be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all of them.

[0033] like Figure 1 As shown, this application provides a flowchart of a method for processing multi-source heterogeneous funding data. The method specifically includes the following steps:

[0034] P100: After accessing multi-source heterogeneous funding data, establish a data source registry, which records at least the unique identifier of the institution to which each data source belongs, the update frequency, the timeliness mark, and the connection method, and marks each piece of funding data with the unique identifier of its corresponding data source, thus establishing the basic architecture of data lineage.

[0035] Furthermore, P101: Multi-source heterogeneous fund data, including bank fund data and unilateral payment platform fund data. Among them, bank fund data includes at least fund data from fund investigation and control platforms, counter bank fund data, anti-money laundering fund data, and national anti-fraud platform fund data. Unilateral payment platform fund data includes at least Tenpay fund data, Alipay fund data, and POS fund data.

[0036] Specifically, in judicial practice, funds involved in cases are often dispersed across different financial institutions and payment platforms, forming complex fund transfer networks. The multi-source, heterogeneous fund data accessed by this system covers the most common data source types encountered by judicial authorities in fund supervision. Furthermore, before accessing various financial institutions and payment platforms, this system requires strict legal authorization and procedural approval. Only then can these institutions and payment platforms provide relevant fund data to the dedicated system through secure data transmission channels. Therefore, the access to multi-source, heterogeneous fund data by this system is a compliant data acquisition process based on complete legal authorization, ensuring the legality of the data sources and the validity of the evidence. Bank fund data, as the core of the traditional financial system, includes data from fund tracing and control platforms used for rapid freezing and querying of accounts involved in cases; counter bank data providing detailed transaction records; anti-money laundering data helping to identify suspicious transaction patterns; and data from the national anti-fraud platform specifically targeting fund flows in telecommunications and online fraud cases. Data from unilateral payment platforms covers the main channels of modern electronic payments. Tenpay and Alipay, as mainstream third-party payment platforms, carry a large amount of daily transaction data, while POS data reflects offline consumption and fund transfers. This comprehensive data access design ensures that judicial authorities can obtain a complete picture of fund flows. The purpose of establishing a data source registry is to achieve unified management and dynamic monitoring of various data sources, complying with financial regulatory requirements. Due to significant differences in data update frequencies, access protocols, and data formats among different financial institutions and payment platforms, the data source registry provides the system with a panoramic view of the data sources and a basis for management by recording key information such as the unique identifier of each data source, update frequency, update timestamp, and access protocol configuration. This design not only facilitates the system's automated acquisition of the latest data from various data sources but also allows for timely adjustments to access strategies when data sources change, ensuring the continuity and accuracy of data acquisition. Establishing a data traceability mechanism stems from the stringent requirements of the integrity of the judicial evidence chain. In judicial proceedings, every piece of evidence must have a clear source and a complete transmission path to ensure its legal validity. By marking each piece of data with a unique identifier of its source and the batch in which it was acquired, key information such as the original source, acquisition time, and processing process of each piece of financial data can be accurately recorded, forming a complete data lineage chain. The source registry serves as the anchor point for data lineage, while the lineage mechanism acts as the executor of registry rules. Together, they constitute a trusted data chain that meets financial regulatory requirements. This not only satisfies the stringent requirements of judicial evidence collection but also provides technical support for subsequent data quality tracing, anomaly data location, and evidence validity verification. It ensures that judicial authorities can clearly prove the legality of the source of evidence and the standardization of the processing procedure in court.

[0037] P200: Based on multi-source heterogeneous funding data carrying data source identifiers, identify its data format, perform differentiated parsing, and output standardized columnar storage data. The standardized columnar storage data adopts a columnar organization structure, with each column corresponding to a field. It stores the original data value set of the field in all multi-source heterogeneous funding data, forming source field data, and is associated with field metadata. Field metadata is standardized descriptive information that characterizes the essential attributes and business characteristics of source field data.

[0038] Furthermore, on page 201: Based on multi-source heterogeneous funding data carrying data source identifiers, the data format of each funding data entry is quickly identified by combining preliminary judgment of file extensions and file header feature detection; on page 202: As funding data is structured data, the file encoding format, field delimitation rules, and table header structure are identified. Based on this, each row of data is parsed and split into discrete field values ​​according to the delimitation rules, the data type of each field is inferred, and a multi-column standardized columnar storage format is constructed to simultaneously generate field metadata generated from parsing the structured data; on page 203: Funding data is semi-structured data. Regular expression methods are used to scan the semi-structured data, identify and extract field names and corresponding field values, and infer the data type of each field. The process involves: constructing a multi-column standardized columnar storage data structure and synchronously generating field metadata generated from parsing semi-structured data; P204: Funding data is unstructured data. Visual elements in the unstructured data are identified through document layout analysis technology. Deep semantic parsing is then performed using a locally deployed financial domain-optimized first LLM model to extract predefined key business entities and their relationships. Based on the entity and relationship extraction results, a multi-column standardized columnar storage data structure is dynamically constructed, and field metadata generated from parsing unstructured data is synchronously generated. Visual elements include at least titles, paragraphs, lists, and embedded tables; P205: The source field data stored in each column of the multi-column standardized columnar storage data structure is bound to its synchronously generated field metadata.

[0039] Specifically, the financial data obtained by different financial institutions and payment platforms through their respective access protocols presents various heterogeneous forms, such as bank financial data and data from fund tracing and control platforms, which are typically in JSON format via HTTPS. API real-time push includes structured account information, frozen status, transaction records, and other fields, with relatively standardized data formats; Counter bank data: mostly provided in batch files in CSV or TXT format, transmitted periodically in T+1 mode, file encoding may be GBK or UTF-8, field separators and header formats vary from bank to bank; Anti-money laundering data: usually in XML format, following specific anti-money laundering reporting standards, containing complex nested structures and a large number of customer identity information fields; National Anti-Fraud Platform data: transmitted in encrypted JSON format via a dedicated network, containing sensitive information such as fraudulent account identifiers and fund flow trajectories; Payment platform data formats: Tenpay and Alipay data: mainly obtained through API interfaces in JSON format transaction data, including merchant information, transaction flow, user identifiers, and other fields, but field naming and data structure vary across platforms; POS data: usually provided in ISO8583 message format or converted structured file format, including merchant code, terminal information, transaction amount, card number information, etc. These raw data exhibit significant differences in format, encoding, field naming, and data type. This heterogeneity is the key technical challenge that the adaptive pattern inference engine needs to address in this step. The adaptive pattern inference engine includes an encoding detection module, a delimiter recognition module, a header location module, and a data type inference module. The encoding detection module statistically analyzes the byte frequency distribution characteristics of the first 1024 bytes of the file and performs pattern matching using a pre-defined encoding feature library to automatically identify encoding formats such as UTF-8, GBK, and ASCII. The delimiter recognition module scans the decoded text content line by line, statistically analyzing the frequency and distribution patterns of candidate delimiters such as commas, tabs, semicolons, and vertical lines, and determines the optimal delimiter through a consistency check algorithm. The header location module uses heuristic rules combined with a machine learning classifier to automatically identify the row position of the header by analyzing the character type distribution, repetition, and format consistency characteristics of the first few rows of data. The data type inference module performs regular expression matching and statistical analysis on each column of sample data, and automatically infers the data type of the field as numeric, date, character, or boolean based on a predefined type recognition rule library. Furthermore, the primary task is to identify and analyze the structure and format characteristics of the original files, rather than extracting and processing the actual business data content. The metadata records the technical characteristics of the files, including structured descriptive information such as encoding format, delimiter type, header position, field names, and data types. This information provides the necessary technical parameters and format specifications for subsequent data content parsing and standardization processing.The process of constructing columnar storage data based on these parsing results involves the system dividing the data content of the original file into fields according to the identified delimiters based on the header positions and field names recorded in the metadata. Then, it performs format conversion and standardization according to the inferred data type, ultimately forming columnar storage data. This is the actual business data after format standardization, including specific fund flow records such as account information, transaction amount, transaction time, and counterparty. The specific implementation of establishing dynamic mapping relationships through unique identifiers involves the system assigning a globally unique file identifier to each original data file, and simultaneously assigning the same association identifier to the corresponding metadata record and columnar storage data record. This identifier association mechanism ensures that any columnar storage data can be accurately traced back to its corresponding metadata information, thereby achieving complete traceability of the data processing process. Metadata is standardized descriptive information that characterizes the essential attributes and business characteristics of source field data. The essential attributes refer to the technical characteristics of the field, including basic features such as field name, data type, length limit, precision requirements, non-null constraints, value range, and uniqueness. Business characteristics refer to the semantic and application features of the field in financial business, including financial professional information such as business meaning identifier, fund attribute classification, regulatory reporting requirements, sensitive information level, data subject domain affiliation, timeliness markers, and cross-system mapping relationships. These metadata together constitute a comprehensive description of the source field. The first LLM model training optimized for the financial domain was conducted using a high-quality training dataset built from financial business documents (including contracts, financial statements, regulatory documents, etc.). Domain experts labeled key business entities (such as accounts, amounts, and terms) and their relationships (such as ownership and constraints). A general LLM foundation model was adopted, and domain adaptation was achieved through a two-stage training process. The first stage involved incremental pre-training using financial corpora (tasks included masked language modeling and financial terminology prediction) to optimize the model's understanding of professional terminology and business logic. The second stage involved supervised fine-tuning based on labeled data, training the model to perform entity recognition, relation extraction, and table reconstruction tasks. The loss function considered both entity accuracy and relation inference consistency. A dynamic curriculum learning strategy was employed during training, gradually increasing data complexity (from structured tables to complex document layouts) and introducing adversarial examples to enhance robustness. Finally, the model was deployed after undergoing financial knowledge graph alignment evaluation (such as entity link accuracy and relation inference F1 score) and business rule compliance verification. Its output, in conjunction with document layout analysis technology, drove the standardized conversion of unstructured data to columnar storage. The entire process retained complete audit records of training data sources, labeling specifications, and model versions.Document layout analysis technology automatically identifies visual elements and their hierarchical relationships in unstructured documents using computer vision and deep learning. The processing steps are as follows: First, the input document undergoes image preprocessing (such as binarization and skew correction). Then, a pre-trained neural network model (such as YOLO or DBNet) detects the bounding boxes and type labels of elements such as titles, paragraphs, and tables. Next, optical character recognition (OCR) is used to extract the text content within each element. Finally, based on the spatial position of the elements (such as indentation and alignment) and style features (such as font and font size), a document logical structure tree is constructed, outputting structured information with coordinates and semantic labels for subsequent LLM semantic parsing. The entire process retains metadata such as element position and recognition confidence.

[0040] For structured financial data, the system first identifies the file encoding format (e.g., UTF-8), field delimiting rules (e.g., comma-separated CSV or tab-separated TSV), and header structure. It then parses the data row by row and splits it into discrete field values ​​according to the delimiting rules. Through value pattern analysis (e.g., numbers, dates, strings), it infers the data type (e.g., integer, floating-point, date, etc.) of each field. The parsed data is then stored column-wise to form a standardized columnar storage structure. Simultaneously, it automatically generates field metadata containing attributes such as field name, data type, length constraints, business description (e.g., "transaction amount"), and data source. Finally, it establishes column indexes to achieve two-way binding between each column of source data and its corresponding metadata. In structured financial data processing, The system builds column indexes by creating a unique column identifier (such as a hash value or an auto-incrementing ID) for each field. This identifier is embedded in both the column header descriptor of the columnar storage data and the metadata storage system, forming a two-way pointer association: at the columnar storage level, the header of each data column's physical storage block records its metadata identifier; at the metadata database level, each metadata record contains a pointer to the storage location of the corresponding data column, and real-time synchronization is maintained through memory mapping or database foreign key constraints. When accessing any data column, the system can quickly retrieve the associated metadata through the column header identifier, and conversely, when modifying metadata, it can also locate the specific data column through the pointer. This two-way indexing mechanism based on shared identifiers ensures the real-time consistency of data and metadata.

[0041] For semi-structured financial data (such as JSON or log text), the system scans the data content using regular expressions, identifies the matching pattern between field names and corresponding values ​​(e.g., "amount": "100.00"), extracts key-value pairs and infers the data type of the values ​​(e.g., recognizing "100.00" as a floating-point number), constructs standardized columnar storage (e.g., using keys as column names and values ​​as column data), and synchronously generates field metadata containing information such as field names, data types (e.g., "transaction amount: float"), business rules (e.g., "positive numbers represent income"), and data extraction paths (e.g., "JSON.$.amount"), and associates each column of data with its metadata through hash mapping. The specific process of the system associating data columns with metadata through hash mapping is as follows: First, for each extracted... The system calculates a unique hash value (such as MD5 or SHA-1) for each field name (e.g., "amount"), and uses this hash value as both the physical identifier of the data column in the columnar storage and the primary key of the metadata record. When building the columnar storage, the system maintains a hash table in memory, using the field hash value as the key to store the memory address or disk location pointer of the corresponding data column. Simultaneously, the same hash value is used as the primary key to store all associated attributes (data type, business rules, etc.) in the metadata database. When a specific field needs to be accessed, the system first calculates the hash value for the field name in real time, quickly locates the data column's storage location through the hash table, and synchronously retrieves the corresponding descriptive information from the metadata database. This bidirectional lookup mechanism based on consistent hashing avoids the performance overhead of string matching while ensuring accurate association between data and metadata.

[0042] For unstructured financial data (such as PDF contracts or scanned invoices), the system first identifies visual elements (such as locating table areas and paragraph text) through layout analysis. Then, it calls the first LLM model optimized for the financial field to parse semantics, extracting predefined entities (such as "account number" and "transaction date") and relationships (such as "payer-payee"), dynamically generating columnar storage (each column corresponds to a business entity such as "amount"). Simultaneously, it generates field metadata containing extended attributes such as entity type (such as "account number: string"), business meaning (such as "invoice serial number"), semantic confidence, and original location (such as "page 3, table row 2 of PDF"). Finally, it achieves the association mapping between column data and metadata through entity IDs. The generation and association mapping of entity IDs are achieved through the following mechanism: The system first assigns a composite entity ID to each business entity (such as "account number") extracted by the LLM model. This ID consists of three parts—document uniqueness, document uniqueness, and other characteristics. A globally unique identifier is synthesized using the UUID algorithm: an identifier (such as the hash value of a PDF file), an entity type code (such as "ACCT_NO"), and the entity's sequence number in the document (such as "003"). When constructing columnar storage, the system uses this entity ID as the physical name of the data column (such as "COL_ACCT_NO_3FE82A"), and simultaneously creates a record in the metadata system with this entity ID as the primary key, storing all extended attributes. To achieve associative mapping, the system maintains an inverted index in memory, where the entity ID serves as the key, pointing to the physical location in the columnar storage and the record address in the metadata system, respectively. When processing data access requests, the system can directly locate the specific column data through the entity ID, or it can retrieve the associated entity ID and its metadata by reversing the process using document coordinates (such as "page 3, table, row 2"). This index structure based on composite IDs effectively solves the problem of entity location and cross-system association in unstructured data.

[0043] P300: Traverse the columnar storage data and its related field metadata, extract statistical features, and generate a data feature description file containing field statistical features and metadata information. Based on this file, perform field semantic recognition and classification.

[0044] By analyzing field metadata and matching it with a financial domain dictionary, fields with significant statistical characteristics and similar business attributes are grouped into the same category. For the remaining unclassified fields, a locally deployed, financial domain-optimized second LLM model is used to parse the true business meaning of the field metadata. The parsing results are then mapped onto a dynamic header-fused knowledge graph to obtain the association between unclassified fields and business entities in the graph.

[0045] Based on the above field classification results and semantic recognition results, a set of mapping relationships from source field data to enterprise-level standardized data models is generated by combining the rule engine template. The rule engine template includes field merging rules, field splitting rules, and field calculation rules.

[0046] Furthermore, P301: Scan the columnar storage data and its related field metadata, extract field names, data types and key statistical features, including: numerical distribution, cardinality, and null value rate; P302: Extract field source information, data type definition and field annotation from the field metadata, and integrate the above statistical features and metadata information into a data profile containing field-level feature descriptions.

[0047] Specifically, a complete data feature extraction and semantic understanding process is established in P300. This process first traverses the standardized columnar data storage and performs comprehensive statistical feature extraction operations. As described in detail in P301, the system scans and analyzes each data column row by row, extracting field names, data types, and key statistical features (including numerical distribution characteristics, cardinality ratio, and null value rate). In P302, it further extracts descriptive information such as field source information, type definitions, and annotations from metadata. These statistical features and metadata information are integrated to form a detailed data profile, providing a foundation for subsequent analysis. Next, the system performs field semantic recognition and classification based on this data profile. This process involves two technical paths: First, traditional feature matching methods are used. By analyzing the statistical features in the data profile and combining them with the system's built-in financial domain professional dictionary, fields with significant statistical features and high matching degrees with business terms in the dictionary are identified, and fields with similar business attributes are grouped into the same category. Then, for fuzzy fields that cannot be classified using the above methods, the system introduces a locally deployed domain-optimized second LLM model for deep semantic understanding. Semantic parsing: This model takes field names, annotations, and sample data as input. Using deep semantic understanding technology, it parses the true business meaning of the fields and maps the results to a dynamically maintained knowledge graph with fused headers. This graph records the semantic relationships between fields and standard business entities. This mapping establishes connections between unclassifiable fields and business entities in the graph, completing semantic recognition for all fields. Finally, based on the complete field classification and semantic recognition results, the system calls its built-in rule engine to generate rules. This rule engine includes three types of rule templates: field merging rule templates (used to merge multiple synonymous source fields into a single standard field), field splitting rule templates (used to split composite information fields into multiple independent standard fields), and field calculation rule templates (used to generate derived fields through mathematical or logical operations). By analyzing the classification and semantic relationships between fields, the system automatically selects the appropriate rule template and parameterizes the specific transformation rules, forming a complete set of mapping relationships from source fields in metadata to the enterprise-level standardized data model. This set covers all transformation logic between fields, providing an execution basis for subsequent data standardization processing.

[0048] Furthermore, on page 303: Statistical features and metadata information for each field are extracted from the data profile. A pre-built financial domain dictionary is invoked, employing a two-stage matching strategy: On page 304: In the first stage, statistical feature similarity calculation is used to identify field groups with similar numerical distributions, cardinality ratios, and null value rates. On page 305: In the second stage, semantic matching is performed between field names and the financial domain dictionary to calculate the similarity between field names and standard terms in the dictionary. On page 306: Combining statistical feature similarity and semantic matching, a comprehensive similarity score is calculated between fields. Based on the comprehensive similarity score, a clustering algorithm is used to group fields with similarity scores higher than a threshold into the same business category, forming the field classification results.

[0049] Specifically, this embodiment implements a field business classification method based on multidimensional feature analysis in steps P303 and P304. This method first extracts complete feature information for each field from the data profile generated in step P300, including statistical features (numerical distribution, cardinality ratio, null value rate, etc.) and metadata information (field name, data type, field comments, etc.). Simultaneously, it loads a pre-built financial domain professional dictionary containing standardized financial business terms and their classification information. Then, the system executes a two-stage matching strategy. In stage P303-1, statistical feature similarity is calculated. Specifically, the numerical distribution features of each field (including maximum, minimum, average, median, standard deviation, quantiles, etc.), cardinality ratio (the ratio of unique values ​​to the total number of records), and null value rate (the ratio of null values ​​to the total number of records) are constructed as multidimensional feature vectors. The similarity between these feature vectors is calculated using Euclidean geometry. The system uses distance to measure the statistical similarity between fields; a smaller distance indicates greater similarity. The system identifies field combinations with statistical feature distances less than a preset threshold. In stage P303-2, the system performs semantic matching between field names and a financial domain dictionary. Specifically, it employs a hybrid character-level and word-level matching strategy. For character-level matching, the system calculates the edit distance (Levenshtein distance) between the field name and the terms in the dictionary. For word-level matching, the system uses N-gram segmentation to convert field names and dictionary terms into word vectors, then calculates the cosine similarity between the word vectors. Finally, the combined result of the two matching methods is taken as the semantic similarity between the field name and the standard term. Next, in step P304, the system constructs a multi-dimensional feature space, using statistical feature similarity and semantic matching similarity as different dimensional coordinates, and calculates the Mahalanobis distance of the field in this multi-dimensional space. The method (Distance) considers the correlation and scale differences between different dimensions, which can effectively measure the comprehensive similarity between fields. The smaller the distance, the higher the comprehensive similarity. Finally, based on the calculated comprehensive similarity, the system applies a hierarchical clustering algorithm to construct a hierarchical tree structure of fields, and truncates the clustering tree at a specific threshold to classify fields in the same cluster into the same business category, forming a complete field classification result. This result accurately reflects the business semantic relationship between fields and provides a foundation for the subsequent generation of mapping rules.

[0050] Furthermore, on page 307: Based on the remaining unclassifiable fields, a second LLM model optimized for the financial domain and deployed locally is used. Based on its natural language understanding technology, combined with financial knowledge base and business rules, the input information is parsed to identify the business semantic essence of the metadata of the remaining unclassifiable fields. The standardized business description and category identifier of the fields are output. The output results are semantically matched with the business entities in the pre-built table header fusion knowledge graph. The similarity between the field business description and the graph entity is calculated, and the matching result with the highest similarity is used as the association mapping between the field and the entity. On page 308: The new field-entity mapping relationship is fed back and updated to the knowledge graph, enabling the graph to dynamically learn and accumulate new business knowledge.

[0051] Specifically, in this embodiment, steps P307 and P308 detail the fuzzy field semantic parsing and knowledge graph update mechanism based on a large language model. For fuzzy fields that cannot be categorized through statistical features and dictionary matching in steps P303 and P304, the system first constructs a structured input containing field names, annotations, and sample data values. This input is then fed into a locally deployed financial domain optimized second LLM model for deep semantic parsing. This model is a specialized model obtained through a multi-stage training process. The training process includes pre-training with domain knowledge enhancement based on a large-scale general pre-trained language model, using financial industry professional literature, regulatory documents, and data standards and specifications. Then, supervised fine-tuning is performed using financial institution data dictionaries and field mapping cases. Next, a multi-task learning paradigm is used to improve the model's ability to understand field semantics. Finally, through specialized training for financial data scenarios and knowledge distillation techniques, a professional model suitable for local deployment is formed. After receiving field information, the model, based on its natural language understanding technology and combined with financial knowledge and business rules integrated within the model, analyzes the field's naming pattern, content features, and contextual relationships to identify the business semantics behind the field. Essentially, the system generates standardized business descriptions and category identifiers. It then semantically matches the business descriptions output by the model with business entities in a pre-built header fusion knowledge graph. This knowledge graph is a structured knowledge base storing the mapping relationships between fields and business entities, containing entity nodes (such as business concepts like "account identifier" and "transaction amount") and relationship edges (such as semantic associations like "equivalent to" and "subcategory"). The system identifies the business entity with the highest similarity by calculating the cosine similarity between the field's business description and the semantic vector representations of each entity in the graph, and uses this entity as the standard mapping target for the field. In step P308, the system feeds back and persistently stores this newly discovered field-entity mapping relationship in the header fusion knowledge graph. It employs graph database incremental update technology to ensure that the newly added mapping relationships remain consistent with existing knowledge, while triggering the knowledge graph's self-optimization mechanism, including relationship propagation algorithms and conflict detection algorithms. This enables the knowledge graph to dynamically learn and accumulate new business knowledge, forming a continuously evolving intelligent mapping system. This mechanism significantly improves the system's ability to process new data sources, providing a more comprehensive semantic foundation for subsequent data fusion and mapping rule generation.

[0052] Furthermore, on page 309: The predefined enterprise-level standardized data model is read, and the rule engine is initialized. This involves setting the standardized fields defined in the enterprise-level standardized data model as target fields, including: field name, data type, business meaning, and format specifications. On page 310: Based on the above field classification and semantic recognition results, source fields with the same business meaning are mapped to a single target field. A field merging rule template is applied to generate merging rules containing source field selection strategies, data format conversion, and conflict handling logic. On page 311: A source field containing multiple business information is mapped to multiple target fields. A field splitting rule template is applied to generate merging rules containing delimiters. Splitting rules for character recognition, content extraction, and format validation; P312: Analyze and identify the target fields that need to be calculated, apply field calculation rule templates, and generate calculation rules containing mathematical expressions, conditional judgments, and function calls; P313: Perform dependency analysis on the generated rules, organize the generated field merging rules, field splitting rules, and field calculation rules into a unified mapping rule set. This set contains the complete transformation logic from source fields to target fields, and persists this rule set to the rule base to provide an execution basis for data transformation. Among them, dependency analysis is performed based on the logical relationships between rules and data processing dependencies.

[0053] Specifically, this embodiment implements an automatic mapping rule generation mechanism based on a rule engine. First, in step P309, a predefined enterprise-level specification data model is read. This specification defines a standardized set of fields and their data types, constraints, business attributes, and other metadata in a structured form. The standardized fields are set as target fields. The data types, constraints, business attributes, and other metadata correspond to field names (e.g., "account_identifier", "transaction_amount"), data types (e.g., numeric, text, date), business meanings (e.g., "account identifier", "transaction amount"), and format specifications (e.g., numeric precision, date format). The system initializes its built-in rule engine, which includes various rule templates and rule generation algorithms to provide technical support for subsequent mapping rule generation. In step P310, based on the field classification and semantic recognition results generated in steps P303-P308, the system identifies multiple source fields with the same business meaning but from different data sources (such as "account number," "account number," and "card number" from different banking systems). The system automatically generates field merging rules using the field merging rule template. These rules include a priority selection strategy for source fields (based on data quality and integrity), data format standardization conversion (such as unified account number format), and potential conflict handling logic (such as the handling method when multiple values ​​exist simultaneously), ensuring that multiple source fields... The system can accurately merge data into a single standard field within the standardized model. In step P311, the system analyzes and identifies composite fields containing multiple business information (such as "Name_ID Number", "Address-Contact Number", etc.), and generates field splitting rules using a field splitting rule template. These rules intelligently analyze field content characteristics, automatically identify delimiter positions (such as "_", "-", or fixed-length delimiters), and design content extraction strategies (such as regular expression extraction) and format validation logic (ensuring that the extracted content meets the requirements of the target field), thus achieving the splitting and conversion of composite fields into multiple standard fields. In step P312, the system automatically generates appropriate mathematical expressions, conditional judgments, and function calls based on the following four key information sources. Rules: First, the system analyzes the target field definitions in the enterprise-level standardized data model, including the field's business meaning, data type, and calculation attribute identifier. For example, when "Transaction Amount in Local Currency" is marked as a calculation field, the system will automatically identify its derived attributes. Second, the system extracts calculation logic templates related to the field from the preset financial business rule base. These templates contain specific calculation paradigms in the financial field, such as exchange rate conversion formulas, risk scoring algorithms, or interest calculation methods. Third, the system analyzes the source field set that has been mapped and identifies the combination of input fields that can support the calculation of the target field. For example, the system will identify that "Transaction Amount" and "Transaction Currency" can be combined with the "Exchange Rate Table" to calculate "Transaction Amount in Local Currency".Fourth, the system references computational patterns from successful historical mapping cases and applies computational rules validated in similar scenarios to the current context through pattern matching and transfer learning techniques. Based on this information, the system constructs complete calculation rules that include necessary mathematical expressions (such as arithmetic formulas for multiplying amount by exchange rate), conditional judgments (such as selecting different calculation paths based on transaction type), and function calls (such as date format conversion and exchange rate lookup functions). These rules are structured into executable logical expressions to ensure that derived fields can be calculated accurately and remain consistent with the enterprise-level data model. Finally, in step P313, the system systematically analyzes all generated rules to identify dependencies between them, specifically including: data dependencies between rules: when the output of one rule is the input of another rule, the former must be executed first. For example, if a field splitting rule needs to process intermediate fields generated by a field merging rule, the merging rule must be executed first; logical relationships between fields: the processing logic of some fields depends on the existence of other fields. For example, the rule for calculating the "transaction amount in local currency" depends on the "transaction amount" and "exchange rate" fields being correctly mapped; processing complexity order: simple one-to-one mappings are usually executed first, followed by complex splitting and merging operations; resource optimization considerations: rules that may filter out a large amount of data are executed first, while resource-intensive rules are executed later. By identifying the pre- and post-execution relationships between different rules through their execution order dependencies, the system determines the optimal execution order and avoids data conversion errors or performance issues caused by improper execution order. In practical applications, priority sorting and dependency analysis is an efficient rule scheduling mechanism. By analyzing the logical relationships and data flow dependencies between different mapping rules, it constructs the optimal rule execution sequence. First, it addresses the data dependencies between rules. When rule B needs to process the output of rule A, the system automatically ensures that rule A is executed first. For example, when processing composite account information, field splitting rules need to wait for field format unification rules to complete. Second, it addresses the business logic relationships between fields. For example, the rule for calculating "risk score" must wait for the mapping of the "transaction frequency" and "transaction amount" fields to be completed before execution. Third, it addresses the gradient sorting based on processing complexity. The system prioritizes simple direct mappings (such as "account" → "account_id"), then processes complex mappings that require merging or splitting (such as merging multiple address fields), and finally executes derived rules involving multi-field calculations (such as calculating the transaction risk index).Fourthly, resource utilization is optimized by prioritizing rules that reduce subsequent data processing volume. Specifically, when processing cross-border transaction data, the system first executes currency identification rules, then exchange rate lookup rules, and finally amount conversion rules. This intelligent sorting mechanism effectively avoids rule conflicts and circular dependencies, ensuring the efficient and reliable execution of complex data transformation processes and significantly improving the accuracy and performance of heterogeneous financial data standardization processing. Finally, all optimized merging, splitting, and calculation rules are organized into a structured set of mapping rules. This set contains the complete transformation path and execution logic from source data to an enterprise-level standardized model, and is persistently stored in the system rule base. This provides executable rules for subsequent data standardization transformations, significantly improving the automation and standardization quality of data processing.

[0054] P400: Based on a set of mapping relationships, it transforms the source field data of multi-source heterogeneous financial data into an enterprise-level standardized data model and performs standardization processing, including data preprocessing to improve data purity, converting financial and digital data into digital formats, and standardizing offsetting identifiers, payment directions, and currency types.

[0055] Furthermore, P401: Based on the mapping relationship set, perform field-level transformation on multi-source heterogeneous funding data, converting each field in the source data into the corresponding target field in the enterprise-level standardized data model according to the mapping rules, generating standardized funding data that conforms to enterprise-level standards; P402: Initiate a data preprocessing verification mechanism on the transformed funding data, identifying spaces, carriage returns, and tabs in the source field data by scanning each field, and implementing a data abnormal character defense mechanism, ensuring that the data does not contain abnormal characters that may affect subsequent processing through character encoding detection and illegal character filtering; P403: Perform standardized data type conversion, uniformly converting funding and numeric data in the source field data into standard numerical formats, ensuring consistency in decimal places and numerical precision; P404: Perform specific content processing, including unifying the offsetting flag to a standard encoding format, converting the payment and receipt flags into unified loan and borrowing flags, and converting different currency codes into international standard currency codes.

[0056] Specifically, this embodiment implements a comprehensive data standardization and preprocessing mechanism in steps P401 to P404. As a direct application of the mapping rule generation in P600, the system first calls the rule engine in step P401 to load the mapping rule set generated in P313, and performs a structured transformation operation on the multi-source heterogeneous funding data. This process applies field merging rules (unifying multiple similar fields into standard fields), field splitting rules (decomposing composite fields into multiple standard fields), and field calculation rules (generating derived fields based on source fields) sequentially according to the transformation logic defined in the rule set, transforming the original heterogeneous funding data. To conform to the unified structure of the enterprise-level standardized data model, the system first loads a predefined set of mapping relationships. This set contains the correspondence rules between source fields and target fields in the enterprise-level standardized data model. For each source data record in the multi-source heterogeneous funding data, the system performs field-level transformation operations according to the mapping relationships. This includes aggregating the original fields in the source data into target fields according to merging rules, decomposing them into multiple target fields according to splitting rules, or generating derived fields through calculation rules. During this process, the original information of the source fields (including field names, data types, data source system identifiers, and business meaning descriptions) is fully preserved as key metadata. The data is retained and linked to the target field to ensure the traceability of the conversion process. In step P402, the system initiates a multi-layered data preprocessing and verification mechanism for the structured financial data. First, an efficient character scanning algorithm is used to detect and identify various redundant characters in the data values ​​field by field, including but not limited to leading spaces, trailing spaces, embedded spaces, carriage returns (CR), line feeds (LF), tabs, and other non-business-related format characters. At the same time, the system deploys a data anomaly character defense mechanism, which identifies possible illegal characters, control characters, or special characters through Unicode encoding range detection and character frequency analysis. The system processes and filters or replaces data according to a preset security policy to ensure that the data does not contain any abnormal characters that may affect subsequent processing. In step P403, the system performs standardized data type conversion specifically for the characteristics of financial data, focusing on precise control of financial data, including unifying the number of decimal places (usually retaining two decimal places for financial transaction amounts), unifying numerical precision (ensuring accurate representation of large transactions), unifying scientific notation conversion (unifying different representations into a common numerical format), and unifying thousands separator processing (removing or uniformly adding thousands separators), ensuring consistency in format and precision for all numerical data.Finally, in step P404, the system performs specialized standard processing on specific content in financial transactions. This includes unified coding of offsetting transaction identifiers (unifying the offsetting, cancellation, and reversal identifiers from different systems into standard codes), standardization of payment and receipt identifiers (converting different representations such as income / expenditure and debit / credit into unified debit / credit identifiers), international standard conversion of currency codes (unifying different representations such as RMB, CNY, etc., into ISO 4217 standard currency codes), and standard mapping of transaction type codes (mapping the proprietary transaction type codes of various institutions into unified standard types). These specialized processes ensure that the converted financial data not only conforms to the standardized model in structure but also achieves a high degree of consistency and standardization in content.

[0057] P500: The standardized financial data is cleaned to further optimize data quality.

[0058] Furthermore, P501: Clustering algorithms are used to perform similarity analysis on the standardized financial data, merging or marking duplicate or similar records to obtain deduplicated standardized financial data; P502: Based on the deduplicated standardized financial data, the account identifier values ​​are extracted from the source data from the fields marked as account identifiers in the field metadata. All historical transaction records of the same account are aggregated based on the account identifiers to form an account-level dataset. By analyzing the temporal logic and business rules between related fields within the account-level dataset, missing or abnormal records are identified. Supplementary values ​​are generated based on the statistical characteristics of historical data of the same account to repair missing or abnormal records; P503: Data cleaning algorithms combined with deep learning models are used to perform intelligent quality detection on the repaired standardized financial data, automatically identifying format errors, logical inconsistencies, and numerical anomalies in the data. For the identified erroneous data, automatic correction is performed according to predefined repair rules, and the cleaned standardized financial data is output.

[0059] Specifically, this embodiment implements an advanced data quality optimization and intelligent cleaning mechanism in steps P501 to P503. As a further deepening of the standardization process, the system first implements similarity analysis based on clustering algorithms in step P501, specifically using an improved DBSCAN (density-based spatial clustering with noise) algorithm. This algorithm calculates the similarity distance between financial data records, grouping records with similarity higher than a certain threshold into the same cluster. The system extracts multi-dimensional feature vectors for each record (including key attributes such as transaction amount, transaction time, counterparty, and transaction type), and calculates the similarity using a weighted Jaccard similarity coefficient. The system assesses the similarity between records and adaptively adjusts the distance threshold based on the characteristics of financial data to ultimately identify potential duplicate records or highly similar transaction records, which are then classified and labeled. In step P502, the system applies a context-related complementary algorithm. This algorithm, based on the business logic characteristics of financial transactions, constructs a transaction association network. Through key fields such as account identifiers and transaction reference numbers, it connects related records in the same account's transaction sequence or the same transaction chain. Then, it applies temporal logic reasoning techniques to analyze the temporal continuity, amount balance, and transaction logic in the transaction sequence, thereby identifying records that may have missing or abnormal data. Once such records are found, the system... The system will supplement relevant transaction information within the context, such as supplementing missing transaction times based on counterparty transaction records, or correcting abnormal amounts based on the correspondence between transfer in and transfer out records. In step P503, the system deploys a deep learning-based data cleaning model. This model employs an architecture combining a bidirectional long short-term memory network (Bi-LSTM) optimized for financial data with an attention mechanism. This deep learning model has been trained on a large amount of real financial data and can capture the complex pattern characteristics of normal and abnormal transactions. The system inputs the preprocessed financial data into the model, evaluates the probability of anomalies for each record in real time, and identifies possible formats. Quality issues include errors (such as inconsistent date formats), logical inconsistencies (such as loan imbalances), and numerical anomalies (such as unusually large transactions). For detected erroneous data, the system applies predefined repair rules for intelligent correction. These repair rules are a set of expert knowledge rules extracted from financial business specifications and historical data patterns, including date format correction rules (such as converting various non-standard date formats to standard formats), amount unit unification rules (such as automatically identifying and correcting inconsistencies in amount units), field value mapping rules (such as mapping non-standard transaction types to standard types), and missing value derivation rules (such as deducing missing necessary information based on relevant fields).For anomalous data that the rules cannot handle or has low confidence levels, the system will specially mark it and generate a detailed anomaly report for subsequent manual review. Ultimately, the system outputs standardized financial data that has undergone comprehensive quality optimization. This data not only conforms to enterprise standards in format and structure but also achieves a high degree of accuracy and consistency in data quality and business logic, providing a reliable foundation for further data verification and analysis.

[0060] P600: Perform time-series continuity, data integrity, and data continuity checks on the further optimized funding data to obtain high-quality, traceable, standardized funding data.

[0061] Furthermore, P601: Perform basic rule verification on the standardized funds data after data cleaning according to the financial business logic rules, including the reasonableness of the transaction amount range, the validity of the account format, and the logic of the transaction time; P602: Perform time series verification, by analyzing the timestamp sequence of each transaction in the same account, to detect whether there is time reversal, time jump, or time repetition, to ensure the logical continuity of transaction records in the time dimension; P603: Perform value range logic verification, to verify whether the values ​​of each field are within a reasonable business range, including that the transaction amount cannot be negative, the changes in the account balance conform to the lending logic, and the counterparty account actually exists; P604: Implement fund data integrity verification, check for missing business fields, verify the completeness of lending and borrowing information in transaction records, and ensure that each fund transfer has complete element information; P605: Perform data continuity verification, analyze the integrity of fund transfers, detect any missing or broken links in the fund transfer process, and ensure a complete tracking path from the source of funds to their final destination; P606: Based on the established data lineage infrastructure, record the lineage information of each fund data that passes verification, including the data source path, processing step records, verification result identifiers, etc., and output high-quality, traceable, standardized fund data.

[0062] Specifically, this embodiment implements a comprehensive data verification and lineage tracing mechanism in step P600 and its subdivisions P601 to P606. As the final quality assurance link for processing multi-source heterogeneous financial data, the system first initiates a rule engine-based verification mechanism in step P601. This rule engine is an inference system specifically designed for financial data verification, consisting of four core components: a rule base, a rule parser, a rule executor, and a result collector. The rule base stores a large number of preset financial business logic rules, which are written in a declarative rule language, possessing high readability and maintainability. The rule parser is responsible for converting the rules into an executable computation graph, and the rule executor efficiently performs rule checks on the data flow, resulting in... The collector aggregates all rule verification results and generates a report. The system uses this rule engine to perform multi-level compliance checks, including basic rule verification such as the reasonableness of transaction amount ranges (e.g., whether a single large transaction exceeds a threshold), account format validity checks (e.g., whether the bank card number conforms to the Luhn algorithm verification), and transaction time logic verification (e.g., whether the transaction time is within the business hours). In step P602, the system performs time series verification, using time series analysis algorithms to check the continuity of the financial data over time. The system first constructs a transaction time series based on account identifiers or transaction reference numbers, and then applies a time series anomaly detection algorithm to analyze the pattern characteristics of the time series, identifying possible time reversals (e.g., transactions that occurred later). The system handles anomalies such as transactions occurring earlier than previous ones, time jumps (e.g., unusually large intervals between transaction times), or time repetitions (e.g., multiple different transactions from the same account at the same time), ensuring the continuity of transaction records across business logic over time. In step P603, the system performs value range logic validation, setting reasonable value ranges and logical constraints for different types of fields. For example, transaction amount fields cannot be negative (except for specific offsetting transactions), account balance changes must conform to the debit-credit balance principle (total debits equal total credits), and counterparty accounts must have valid records in the system. These constraints verify the logical consistency of financial data at the business level. In step P604, the system is implemented... The integrity verification of fund data, based on the essential elements model of financial transactions, checks whether each transaction record contains all necessary field information, such as whether key fields such as transaction account, transaction amount, transaction time, and transaction type are missing. At the same time, it verifies whether the information of the lending and borrowing parties in the transaction is completely recorded to ensure that each fund transfer has complete transaction elements. In step P605, the system performs data continuity verification. By constructing a fund transfer network graph model, it analyzes the complete flow path of funds from the source to the final destination, and detects whether there are any missing or broken links in the fund transfer process. For example, a fund transfer record exists but the transfer-in record is missing, or the fund amount is inconsistent during the flow process, to ensure the integrity of the fund transfer chain and the consistency of the amount.Finally, in step P606, the system, combining the data lineage base established in step P100, completes the lineage information for each verified financial data record. This includes detailed data source paths (original data files and locations), complete processing step records (transformation and cleaning operations performed), and comprehensive verification result identifiers (passed verification items and levels), providing traceability information. In this way, the system ultimately outputs clean, standardized financial data that not only meets high standards in data quality but also possesses complete end-to-end data lineage tracing capabilities. This ensures that any standardized financial data record can be traced back to its original source and processing history, meeting the stringent requirements of judicial auditing and regulatory compliance.

[0063] like Figure 2 As shown, based on the same inventive concept as the multi-source heterogeneous funding data processing method in the foregoing embodiments, this application provides a multi-source heterogeneous funding data processing system, the system comprising:

[0064] The access fund data module 11 is used to access multi-source heterogeneous fund data, establish a data source registry, record at least the unique identifier of the institution to which each data source belongs, update frequency, timeliness mark and connection method, and mark the unique identifier of its corresponding data source for each piece of fund data, thus establishing the basic architecture of data lineage.

[0065] The fund data parsing module 12 is used to identify the data format of multi-source heterogeneous fund data carrying data source identifiers, perform differentiated parsing, and output standardized columnar storage data. The standardized columnar storage data adopts a columnar organization structure, with each column corresponding to a field. It stores the original data value set of the field in all multi-source heterogeneous fund data to form source field data, which is associated with field metadata. Field metadata is standardized descriptive information that characterizes the essential attributes and business characteristics of source field data.

[0066] The financial data fusion module 13 is used to traverse the columnar storage data and its related field metadata, extract statistical features, and generate a data feature description file containing field statistical features and metadata information. Based on this file, field semantic recognition and classification are performed.

[0067] By analyzing field metadata and matching it with a financial domain dictionary, fields with significant statistical characteristics and similar business attributes are grouped into the same category. For the remaining unclassified fields, a locally deployed, financial domain-optimized second LLM model is used to parse the true business meaning of the field metadata. The parsing results are then mapped onto a dynamic header-fused knowledge graph to obtain the association between unclassified fields and business entities in the graph.

[0068] Based on the above field classification results and semantic recognition results, a set of mapping relationships from source field data to enterprise-level standardized data models is generated by combining the rule engine template. The rule engine template includes field merging rules, field splitting rules, and field calculation rules.

[0069] The fund data standardization processing module 14 is used to convert the source field data of multi-source heterogeneous fund data into an enterprise-level standardized data model based on the mapping relationship set, and to perform standardization processing, including data preprocessing to improve data purity, converting fund and digital data into digital format, and standardizing the offsetting identifier, payment direction and currency type.

[0070] The Funds Data Optimization Module 15 is used to clean the standardized funds data and further optimize the data quality.

[0071] The fund data verification module 16 verifies the time sequence continuity, data integrity, and data continuity of the further optimized fund data to obtain high-quality, traceable, and standardized fund data.

[0072] Through the foregoing detailed description of the processing method for multi-source heterogeneous financial data, those skilled in the art can clearly understand the multi-source heterogeneous financial data processing system in this embodiment. As for the apparatus disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and relevant parts can be referred to the method section description.

[0073] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing multi-source heterogeneous financial data, characterized in that, The method includes: After accessing multi-source heterogeneous funding data, a data source registry is established to record at least the unique identifier of the institution to which each data source belongs, the update frequency, the timeliness mark, and the connection method. Each piece of funding data is also marked with the unique identifier of its corresponding data source, thus establishing the basic architecture of data lineage. Based on multi-source heterogeneous funding data carrying data source identifiers, the data format is identified, differential parsing is performed, and standardized columnar storage data is output. The standardized columnar storage data adopts a columnar organization structure, with each column corresponding to a field. It stores the original data value set of the field in all multi-source heterogeneous funding data, forming source field data, and is associated with field metadata. Field metadata is standardized descriptive information that characterizes the essential attributes and business characteristics of source field data. The system iterates through the columnar data and its associated field metadata, extracts statistical features, and generates a data feature description file containing field statistical features and metadata information. Based on this file, it performs field semantic recognition and classification. By analyzing field metadata and matching it with a financial domain dictionary, fields with significant statistical characteristics and similar business attributes are grouped into the same category. For the remaining unclassified fields, a locally deployed, financial domain-optimized second LLM model is used to parse the true business meaning of the field metadata. The parsing results are then mapped onto a dynamic header-fused knowledge graph to obtain the association between unclassified fields and business entities in the graph. Based on the above field classification results and semantic recognition results, a set of mapping relationships from source field data to enterprise-level standardized data models is generated by combining the rule engine template. The rule engine template includes field merging rules, field splitting rules, and field calculation rules. Based on the mapping relationship set, the source field data of multi-source heterogeneous financial data is transformed into an enterprise-level standardized data model and standardized processing is performed, including data preprocessing to improve data purity, converting financial and digital data into digital format, and standardizing the offsetting identifier, payment direction and currency type. The standardized financial data will be cleaned to further optimize data quality; The optimized funding data is then verified for temporal continuity, data integrity, and data continuity to obtain high-quality, traceable, standardized funding data.

2. The method for processing multi-source heterogeneous financial data according to claim 1, characterized in that, Based on multi-source heterogeneous funding data carrying data source identifiers, the data format is identified, differential parsing is performed, and standardized columnar storage data is output, including: Based on multi-source heterogeneous financial data carrying data source identifiers, the data format of each piece of financial data can be quickly identified by combining preliminary judgment of file extension and detection of file header features. As structured data, financial data is identified by its file encoding format, field delimitation rules, and table header structure. Based on this, each row of data is parsed and split into discrete field values ​​according to the delimitation rules. The data type of each field is inferred, and a multi-column standardized columnar storage format is constructed to store the data. Simultaneously, the field metadata generated by parsing the structured data is generated. The financial data is semi-structured. The semi-structured data is scanned using regular expressions to identify and extract the field names and corresponding field values, infer the data type of each field, construct a multi-column standardized columnar storage format, and synchronously generate the field metadata generated by parsing the semi-structured data. The financial data is unstructured. Visual elements in the unstructured data are identified through document layout analysis technology. Deep semantic parsing is performed by combining the first LLM model optimized for the financial field deployed locally. Predefined key business entities and their relationships are extracted. Based on the extraction results of entities and relationships, multi-column standardized columnar storage data is dynamically constructed. Field metadata generated from parsing unstructured data is generated simultaneously. Visual elements include at least titles, paragraphs, lists and embedded tables. Bind the source field data stored in each column of the multi-column standardized columnar storage data to the field metadata generated synchronously.

3. The method for processing multi-source heterogeneous financial data according to claim 2, characterized in that, Iterate through the columnar storage data and its related field metadata, extract statistical features, and generate a data feature description file containing field statistical features and metadata information, including: Scan columnar storage data and its related field metadata to extract field names, data types and key statistical features, including: numerical distribution, cardinality and null value rate. Extract field source information, data type definitions, and field annotations from the field metadata, and integrate the above statistical features and metadata information into a data profile containing field-level feature descriptions.

4. The method for processing multi-source heterogeneous financial data according to claim 3, characterized in that, Through field metadata analysis and matching with financial domain dictionaries, fields with significant statistical characteristics and similar business attributes are grouped into the same category, including: The statistical features and metadata of each field are extracted from the data profile, and a two-stage matching strategy is adopted by calling a pre-built financial professional dictionary. The first stage identifies field groups with similar numerical distributions, cardinality ratios, and null value rates through statistical feature similarity calculations. The second stage involves semantic matching of field names with a financial domain dictionary to calculate the similarity between field names and standard terms in the dictionary. By combining statistical feature similarity and semantic matching, a comprehensive similarity score is calculated between fields. Based on the comprehensive similarity score, a clustering algorithm is used to group fields with similarity higher than a threshold into the same business category, thus forming the field classification results.

5. The method for processing multi-source heterogeneous financial data according to claim 3, characterized in that, For the remaining unclassifiable fields, a second LLM model optimized for the financial sector, deployed locally, is used to analyze the true business implications of the fields. The analysis results are then mapped to a dynamic header fusion knowledge graph to obtain the associations between the unclassifiable fields and business entities in the graph, including: Based on the remaining unclassifiable fields, a second LLM model optimized for the financial field is deployed locally. Based on its natural language understanding technology, combined with financial knowledge base and business rules, the input information is parsed to identify the business semantic essence of the metadata of the remaining unclassifiable fields. The standardized business description and category identifier of the fields are output. The output results are semantically matched with the business entities in the pre-built table header fusion knowledge graph. The similarity between the field business description and the graph entity is calculated, and the matching result with the highest similarity is used as the association mapping between the field and the entity. The new field-entity mapping relationship is fed back and updated to the knowledge graph, enabling the graph to dynamically learn and accumulate new business knowledge.

6. The method for processing multi-source heterogeneous financial data according to claim 4 or 5, characterized in that, Based on the above field classification and semantic recognition results, a set of mapping relationships from source fields in the metadata to the enterprise-level standardized data model is generated using the rule engine template. The rule engine template includes field merging rules, field splitting rules, and field calculation rules, including: Read the predefined enterprise-level standardized data model and initialize the rule engine. In this process, set the standardized fields defined in the enterprise-level standardized data model as target fields, including: field name, data type, business meaning and format specification. Based on the above field classification results and semantic recognition results, source fields with the same business meaning are mapped to a target field. The field merging rule template is applied to generate merging rules that include source field selection strategies, data format conversion, and conflict handling logic. Map a source field containing multiple business information to multiple target fields, apply a field splitting rule template, and generate splitting rules that include delimiter recognition, content extraction, and format validation; Analyze and identify the target fields that need to be calculated, apply field calculation rule templates, and generate calculation rules that include mathematical expressions, conditional judgments, and function calls; Dependency analysis is performed on the generated rules. The generated field merging rules, field splitting rules, and field calculation rules are organized into a unified mapping rule set. This set contains the complete transformation logic from source fields to target fields. This rule set is persisted to the rule base to provide the basis for data transformation. The dependency analysis is performed based on the logical relationships between rules and the data processing dependencies.

7. The method for processing multi-source heterogeneous financial data according to claim 1, characterized in that, Based on the mapping relationship set, the source data fields of multi-source heterogeneous funding data are transformed into an enterprise-level standardized data model and standardized, including: Based on the set of mapping relationships, field-level transformation is performed on multi-source heterogeneous funding data. Each field in the source data is converted into the corresponding target field in the enterprise-level standardized data model according to the mapping rules, generating standardized funding data that conforms to enterprise-level standards. A data preprocessing verification mechanism is initiated for the converted financial data. By scanning each field, spaces, carriage returns, and tabs in the source field data are identified. At the same time, a data abnormal character defense mechanism is implemented. Through character encoding detection and illegal character filtering, it is ensured that the data does not contain abnormal characters that may affect subsequent processing. Perform standardized data type conversion to convert financial and numeric data in the source field data into a standard numerical format, ensuring consistency in decimal places and numerical precision; Content standardization processing includes: standardizing the reversal identifier to a standard coding format, converting payment and receipt identifiers into unified lending and borrowing identifiers, and converting different currency codes into international standard currency codes.

8. The method for processing multi-source heterogeneous financial data according to claim 7, characterized in that, The standardized financial data will undergo data cleaning to further optimize data quality, including: Clustering algorithms are used to perform similarity analysis on the standardized financial data, and duplicate or similar records are merged or marked to obtain deduplicated standardized financial data. Based on the deduplicated standardized financial data, the field marked as account identifier in the field metadata is extracted from the source data. The account identifier is then aggregated based on the account identifier to form an account-level dataset. By analyzing the temporal logic and business rules between related fields in the account-level dataset, missing or abnormal records are identified. Supplementary values ​​are generated based on the statistical characteristics of the historical data of the same account to repair the missing or abnormal records. By combining data cleaning algorithms with deep learning models, intelligent quality inspection is performed on the repaired standardized financial data. The system automatically identifies format errors, logical inconsistencies, and numerical anomalies in the data. For the identified erroneous data, it automatically corrects them according to predefined repair rules and outputs the cleaned standardized financial data.

9. The method for processing multi-source heterogeneous financial data according to claim 8, characterized in that, The optimized funding data was further validated for time-series continuity, data integrity, and data continuity, including: Based on the financial business logic rules, the standardized funds data after data cleaning is verified by basic rules, including the reasonableness of the transaction amount range, the validity of the account format, and the logic of the transaction time. Perform time series verification by analyzing the timestamp sequence of each transaction in the same account to detect whether there is time reversal, time jump or time repetition, to ensure the logical continuity of transaction records in the time dimension; Perform value range logic validation to verify whether the values ​​of each field are within a reasonable business range, including that the transaction amount cannot be negative, the account balance changes are consistent with the lending logic, and the counterparty's account actually exists. Implement data integrity verification for funds, check for missing business fields, verify the completeness of lending and borrowing information in transaction records, and ensure that each fund transfer has complete information. Perform data continuity verification by analyzing the integrity of fund flow to detect any missing or broken links in the fund flow process, ensuring a complete tracking path from the source of funds to their final destination; Based on the established data lineage infrastructure, the lineage information of each verified financial data is recorded, including the data source path, processing steps, and verification result identifier, to output high-quality, traceable, and standardized financial data.

10. A system for processing multi-source heterogeneous financial data, characterized in that, The system includes: The access fund data module is used to access multi-source heterogeneous fund data, establish a data source registry, record at least the unique identifier of the institution to which each data source belongs, update frequency, timeliness mark and connection method, and mark the unique identifier of its corresponding data source for each piece of fund data, thus establishing the basic architecture of data lineage; The funds data parsing module is used to identify the data format of multi-source heterogeneous funds data carrying data source identifiers, perform differentiated parsing, and output standardized columnar storage data. The standardized columnar storage data adopts a columnar organization structure, with each column corresponding to a field. It stores the original data value set of the field in all multi-source heterogeneous funds data, forming source field data, and is associated with field metadata. Field metadata is standardized descriptive information that characterizes the essential attributes and business characteristics of source field data. The financial data fusion module is used to traverse columnar storage data and its related field metadata, extract statistical features, and generate a data feature description file containing field statistical features and metadata information. Based on this file, field semantic recognition and classification are performed. By analyzing field metadata and matching it with a financial domain dictionary, fields with significant statistical characteristics and similar business attributes are grouped into the same category. For the remaining unclassified fields, a locally deployed, financial domain-optimized second LLM model is used to parse the true business meaning of the field metadata. The parsing results are then mapped onto a dynamic header-fused knowledge graph to obtain the association between unclassified fields and business entities in the graph. Based on the above field classification results and semantic recognition results, a set of mapping relationships from source field data to enterprise-level standardized data models is generated by combining the rule engine template. The rule engine template includes field merging rules, field splitting rules, and field calculation rules. The Funds Data Standardization Processing Module is used to transform source field data of multi-source heterogeneous funds data into an enterprise-level standardized data model based on a set of mapping relationships, and to perform standardization processing, including data preprocessing to improve data purity, converting funds and digital data into digital formats, and standardizing offsetting identifiers, payment directions, and currency types. The funds data optimization module is used to clean the standardized funds data and further optimize the data quality. The funds data verification module is used to verify the time sequence continuity, data integrity, and data continuity of the further optimized funds data, so as to obtain high-quality, traceable, and standardized funds data.

Images