A Data Intelligent Acquisition and Transformation Method Based on Visual Rule Model

Abstract

This invention discloses a data intelligent acquisition and transformation method based on a visual rule model. The method includes: S1, a visual rule modeling layer converts the configured rule chain into a JSON-formatted data body, parses the data body into an AST (Abstract Syntax Tree), maps fields to the AST, and then uses an intelligent transformation engine to convert the mapped fields into executable code, generating an execution plan; S2, an intelligent acquisition layer pulls data from heterogeneous data sources, cleans the data, and distributes it to the intelligent transformation engine for parallel processing; S3, the intelligent transformation engine monitors the data lineage graph in real time, monitors field-level transformation paths, and sends signals to the visual rule modeling layer or the intelligent acquisition layer based on the type of anomaly, to achieve rule chain reconfiguration or data re-acquisition. The advantages are: it enables code-free integration and real-time monitoring of heterogeneous system data, breaking through the challenges of dynamic adaptation, semantic understanding, and data lineage interpretability in fiscal fund management.

Description

Technical Field

[0001] This invention relates to the field of data intelligent processing technology, and in particular to a data intelligent acquisition and transformation method based on a visual rule model. Background Technology

[0002] With the rapid development of the social economy, the scale of fiscal funds has continued to expand, and the flow of funds involves multiple levels of government departments, enterprises, public institutions, and public service sectors. However, the cross-departmental collaborative management of fiscal funds faces severe challenges, specifically manifested in the following ways:

[0003] 1. Data silo problem in heterogeneous systems

[0004] Data structure conflict: The finance department uses the Golden Finance Project (based on Oracle database, with field naming in camelCase), while the project management system uses a collaborative platform (based on MySQL, Highgo database, etc., with field naming in underscores). Due to the difference in naming rules, the system cannot automatically match them.

[0005] Inconsistent business logic: Fund disbursement rules are defined differently in different systems (e.g., "Project progress ≥ 80%" is a numeric field in the Financial Engineering system, but a percentage string in the Project Management System).

[0006] 2. Low efficiency of manual processing

[0007] Cross-system integration is time-consuming: It relies on Excel or scripts for field mapping and format conversion, and a single data integration takes 3-5 working days. Moreover, manual operation is prone to errors (such as omitting the multiplier factor when converting units from "ten thousand yuan to yuan", leading to common-sense issues such as amount errors).

[0008] Delayed rule adjustments: Policy changes (such as changing the fund disbursement conditions from "progress ≥ 80%" to "progress ≥ 70% and audit rating ≥ B") require the redevelopment of SQL / Python / Java scripts, with an average response time of more than 2 weeks.

[0009] 3. Insufficient technical barriers and dynamic adaptation capabilities

[0010] Limitations of ETL tools: Traditional tools require writing XML configuration files, and business users cannot directly participate in rule configuration through a visual interface.

[0011] Lack of semantic understanding: Existing solutions rely on regular expressions to match field names or manual matching, which cannot recognize business semantics (such as the equivalence relationship between "Proj_No" and "Project_ID"), resulting in high mapping error rates and long processing times.

[0012] 4. Data lineage tracing and interpretability deficiencies

[0013] Fragmented data flow: Existing technologies only record table-level data flow and lack field-level transformation path tracking. For example, it is impossible to trace the complete processing chain (such as data cleaning, unit conversion, and rule triggering) of the "amount" field from raw data to fund disbursement instructions.

[0014] Traditional cross-departmental data fusion methods (such as traditional ETL tools, API customization solutions, data middle platform architecture, and open-source rule engines) cannot effectively solve the above problems due to their inherent shortcomings.

[0015] (1) Disadvantages of traditional ETL tools (such as Informatica / Kettle):

[0016] The lack of visual rule modeling capabilities necessitates developers writing complex configuration files.

[0017] Dynamic semantic parsing is not supported; nested conditional judgments must be hard-coded (e.g., multi-level IF-ELSE logic).

[0018] (2) Disadvantages of API customization solutions:

[0019] The development cost of a single interface can be as high as 10 person-days, and it is necessary to maintain compatibility with multiple versions (such as the failure of historical interfaces due to the upgrade of the financial system API).

[0020] Incremental data synchronization relies on a polling mechanism, which has poor real-time performance (delay ≥ 1 hour) and cannot meet the minute-level response requirements for fund disbursement.

[0021] (3) Disadvantages of data middle platform architecture (such as Alibaba DataWorks):

[0022] The implementation cycle is long (≥3 months), and small and medium-sized institutions can hardly afford the resource investment.

[0023] The rules engine is highly coupled with the business system, and dynamic adjustments require downtime for deployment, which affects business continuity.

[0024] (4) Disadvantages of open-source rule engines (such as Drools):

[0025] The rules and logic must be defined in code (such as DRL language), and business personnel cannot edit them directly.

[0026] The lack of intelligent field mapping and data lineage closed-loop feedback results in low efficiency in anomaly localization (average investigation time ≥ 4 hours). Summary of the Invention

[0027] The purpose of this invention is to provide a data intelligent collection and transformation method based on a visual rule model, thereby solving the aforementioned problems existing in the prior art.

[0028] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0029] A data intelligent collection and transformation method based on a visual rule model includes the following steps:

[0030] S1, the visual rule modeling layer, converts the configured rule chain into a JSON format data body, parses the data body into an AST abstract syntax tree, maps the fields of the AST abstract syntax tree, and then uses the intelligent transformation engine to convert the mapped fields into executable code to generate an execution plan;

[0031] S2, the intelligent acquisition layer pulls data from heterogeneous data sources based on full acquisition and / or incremental acquisition, cleans the data, and then distributes it to the intelligent conversion engine for parallel processing;

[0032] S3, the intelligent conversion engine, monitors the data lineage graph in real time, monitors field-level conversion paths, and sends signals to the visualization rule modeling layer or intelligent acquisition layer according to the type of anomaly, so as to realize rule chain reconfiguration or data re-acquisition.

[0033] Preferably, step S1 specifically includes the following:

[0034] S11. Users configure rule chains in the drag-and-drop rule component library of the visual rule modeling layer;

[0035] S12, the dynamic semantic parsing engine of the visual rule modeling layer converts the rule chain into an intermediate representation in JSON format, which includes component types, parameters and connection relationships;

[0036] S13, the dynamic semantic parsing engine of the visualization rule modeling layer further parses the intermediate representation into an AST abstract syntax tree and optimizes and merges redundant nodes;

[0037] S14. The intelligent field mapping module of the visualization rule modeling layer performs field mapping on the AST abstract syntax tree and verifies the mapping error rate.

[0038] S15. The mapped fields are converted into executable code through the dynamic rule compilation module of the intelligent transformation engine, and Spark SQL or Java classes are generated in real time according to the scenario.

[0039] S16. Submit the executable code to the distributed execution framework of the intelligent transformation engine to generate a physical execution plan.

[0040] Preferably, in step S14, the field mapping process of the intelligent field mapping module is as follows:

[0041] S141. Data Input and Preprocessing: Receive source data fields and target structure fields, and standardize them.

[0042] S142. Multi-dimensional feature extraction: Use the BERT model to analyze the contextual similarity between the source data fields and the target structure fields, perform structural feature matching between the source data fields and the target structure fields, and calculate the edit distance of the field names between the source data fields and the target structure fields;

[0043] S143. Similarity Calculation and Decision: Calculate the weighted composite score between the source data fields and each target structure field based on the semantic similarity, structural matching degree, and naming similarity between the source data fields and the target structure fields. When the weighted composite score is greater than or equal to the total score threshold, the source data fields are automatically mapped to the corresponding target structure fields, and the mapping results are verified. If the verification passes, a mapping scheme is generated and a mapping relationship table is output. If the weighted composite score is less than the total score threshold or the mapping result verification fails, a manual intervention interface is triggered, displaying the top three candidate target structure fields.

[0044] S144, Feedback closed-loop optimization: Users manually correct the mapping relationship and store the correction results in the knowledge base to update the training set of the BERT model and improve the subsequent matching accuracy.

[0045] Preferably, step S2 specifically includes the following:

[0046] S21. The multi-source adapter of the intelligent acquisition layer pulls data from heterogeneous data sources;

[0047] S22. For incremental data collection, the incremental data collection strategy module of the intelligent data collection layer will perform dual verification of the pulled data using timestamp and content hash.

[0048] S23, The data cleaning module of the intelligent acquisition layer performs standardization processing on the pulled data;

[0049] S24. The standardized data is partitioned according to the primary key hash and distributed to the cluster nodes of the distributed execution framework for parallel processing.

[0050] Preferably, the intelligent acquisition layer selects either the Flink engine or the Spark engine to pull data from heterogeneous data sources based on the data volume of the link. When the data volume is greater than 0.8TB and the latency is less than 5 minutes, the Flink engine is enabled, and the checkpoint interval is set to 300 seconds. When batch processing of data is selected, the Spark engine is enabled, the number of partitions is fixed at 200, and the database automatically distributes the data to 200 physical sub-tables based on the hash value. For frequently queried fields, local indexes are created to avoid performance degradation in queries.

[0051] Preferably, step S3 specifically includes the following:

[0052] S31, the intelligent conversion engine's conversion process visualization module monitors the data lineage graph in real time and records the field-level conversion path;

[0053] S32. If the field mapping error rate is greater than the preset threshold, an optimization signal is sent to the visualization rule modeling layer to trigger the reconfiguration of the rule chain.

[0054] S33. If the data quality is abnormal, an alarm will be sent to the intelligent acquisition layer to start the data re-acquisition process.

[0055] Preferably, the monitoring process and results of the data pedigree mapping are displayed using a data pedigree tracking interface.

[0056] The data lineage tracing interface includes a data input layer for displaying the original data source, a data processing layer for displaying the conversion logic, an application output layer for displaying the final output, and an external system interaction layer for displaying external interfaces.

[0057] The data lineage tracking interface can dynamically render the data link and automatically draw arrow connections based on the data flow path. Normal and abnormal data links are represented by different colors and linear connection lines, respectively. At the same time, the interface is equipped with node indicator lights, which are used to represent normally operating nodes and abnormal nodes.

[0058] Preferably, the interactive operation flow of the data lineage tracking interface is as follows:

[0059] A1. Node Metadata Viewing: By clicking on a node, you can view the transformation rules, association rules, execution logs, and abnormal data links; for abnormal data links, clicking on the connecting line representing the link will show the reason for the data link abnormality;

[0060] A2. Time Range Filtering: By dragging the timeline or entering the start and end times, the interface dynamically refreshes the data lineage graph, displaying only the data flow path for the corresponding time period; by selecting two different time periods, the difference links can be highlighted.

[0061] A3. Anomaly Handling and Feedback: By clicking on an anomaly node, it can be marked as processed and the corresponding link status can be updated; if manual intervention is required, the anomaly details will be automatically filled in to generate an anomaly work order, which will be pushed to the audit department.

[0062] A4. Exporting Data Lineage Report: By clicking the export button and selecting the export format, you can generate and export a data lineage report.

[0063] Preferably, the exported data lineage report includes a complete data lineage map, metadata of key nodes, and a statistical table of abnormal data links.

[0064] Preferably, the handling process for abnormal data links in the data lineage tracking interface is as follows:

[0065] B1. Anomaly Detection and Alarm: Real-time scanning of data links. If an anomaly is detected, a floating alarm box will pop up on the data lineage tracing interface, indicating the number of abnormal data links detected. An abnormal data link work order will be automatically generated and sent to the relevant business and technical departments.

[0066] B2. Manual Intervention and Correction: Click on the field node in the abnormal data link, manually select the target field, and click Confirm Mapping to correct the mapping relationship; the correction result is stored in the knowledge base for optimizing subsequent intelligent matching models;

[0067] B3. Historical Version Rollback: Click the Historical Version button, select the rule version at a certain point in time, and the data lineage graph of that version will be automatically restored; the data lineage tracking interface uses color highlighting to show the differences between versions.

[0068] The beneficial effects of this invention are: 1. Two-way visualization → code conversion mechanism: It pioneers a "what you see is what you get" rule designer, supporting business personnel or technical support personnel to configure complex logic without coding, while retaining the developer's code-level flexibility. 2. Semantic enhancement model for the financial domain: Based on a domain-pre-trained BERT intelligent mapping algorithm, it solves similar semantic gap problems in traditional solutions. 3. Dynamic policy engine and elastic computing: Combining rule version management (canary release / rollback) with distributed engine self-adaptation, it achieves a minute-level closed loop of "policy change → rule adjustment → execution taking effect". Attached Figure Description

[0069] Figure 1 This is a system architecture diagram of the acquisition and conversion method in this embodiment of the invention;

[0070] Figure 2 This is a flowchart of the visual rule configuration process in an embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the intelligent field mapping algorithm in an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram of the data lineage tracking interface in an embodiment of the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Example

[0074] This embodiment provides a data intelligent collection and transformation method based on a visual rule model. Unlike traditional cross-departmental data fusion methods, this method uses a three-in-one technical framework of "visual rule modeling + semantic intelligent parsing + dynamic strategy engine" to achieve code-free connection and real-time monitoring of heterogeneous system data. This method abandons the "hard-coded" mode of traditional solutions and solves the problems of dynamic adaptation, semantic understanding, and data lineage interpretability in fiscal fund management based on the following core technological breakthroughs.

[0075] like Figure 1 As shown in the diagram, and considering the system architecture, the intelligent data acquisition and transformation method mainly includes three main interactive processes.

[0076] I. Interaction between the Visual Rule Modeling Layer and the Intelligent Transformation Engine

[0077] The visual rule modeling layer converts the configured rule chain into a JSON-formatted data body, parses the data body into an Abstract Syntax Tree (AST), maps fields within the AST, and then uses an intelligent transformation engine to convert the mapped fields into executable code, generating an execution plan. Specifically, this includes the following:

[0078] 1.1 Users configure rule chains (e.g., "Project progress ≥ 80% and audit passed → trigger fund disbursement") in the drag-and-drop canvas of the visual rule modeling layer.

[0079] 1.2 The dynamic semantic parsing engine converts the rule chain into an intermediate representation in JSON format, which includes component types, parameters, and connection relationships.

[0080] 1.3 The intermediate representation is further parsed into an abstract syntax tree (AST) to optimize and merge redundant nodes (such as duplicate field mappings).

[0081] 1.4. The AST is converted into executable code through a dynamic rule compilation module, generating Spark SQL or Java classes according to the scenario.

[0082] 1.5. Executable code is submitted to a distributed execution framework (Spark / Flink) to generate a physical execution plan (such as Spark DAG).

[0083] II. Interaction between the Intelligent Acquisition Layer and the Intelligent Conversion Engine

[0084] The intelligent data acquisition layer pulls data from heterogeneous data sources using full and / or incremental acquisition methods, cleans and processes the data, and then distributes it to the intelligent transformation engine for parallel processing. Specifically, it includes the following:

[0085] 2.1 The multi-source adapter pulls data from heterogeneous data sources (such as Kingdee financial system API, Oracle database).

[0086] Full data collection: Execute SELECT * FROM table on the first connection.

[0087] Incremental data collection: based on dual verification using timestamps and content hashes.

[0088] 2.2 The data cleaning module performs standardization processing on the collected data:

[0089] Fill empty values: Fill numeric fields with 0, and fill character fields with "N / A".

[0090] Anomaly detection: Detect negative amounts and trigger an alarm.

[0091] 2.3 The cleaned data is partitioned by primary key hash and distributed to cluster nodes for parallel processing.

[0092] III. Interaction between the intelligent conversion engine, the visual rule modeling layer, and the intelligent data acquisition layer

[0093] The intelligent transformation engine monitors the data lineage graph in real time, tracks field-level transformation paths, and sends signals to the visualization rule modeling layer or intelligent acquisition layer based on the type of anomaly, enabling rule chain reconfiguration or data re-acquisition. Specifically, this includes the following:

[0094] 3.1 The visualization layer of the conversion process monitors the data lineage graph in real time and records the field-level conversion path (such as "Financial System.Amount → Conversion Engine.Unit Conversion → Project Management.Amount").

[0095] 3.2 If the field mapping error rate is >5%, send an optimization signal to the rule modeling layer to trigger rule chain reconfiguration.

[0096] 3.3 If the data quality is abnormal (e.g., the null value rate exceeds the threshold), send an alarm to the acquisition layer and start the data re-acquisition process.

[0097] In this embodiment, see the attached Figure 2 The specific process of configuring and executing visual rules is as follows:

[0098] 1. Drag and drop components

[0099] (1) User drag-and-drop behavior components, including condition, mapping, trigger class and other components.

[0100] (2) The system generates a JSON data body, including component node IDs, parameters, connection relationships, etc. The generated format is as follows:

[0101] {"node_id":"cond_001","type":"threshold","expr":"progress>=0.8"})

[0102] (3) Optimize the structure when generating the AST.

[0103] (4) Merge duplicate mappings. Identify the same field conversion operation by comparing SHA-256 hashes. For example, if the unit of measurement for this field is "ten thousand yuan" multiple times, but the unit of measurement for this time is "yuan", the system will adjust the AST structure and give a prompt message.

[0104] (5) Eliminate redundant branches and delete perpetually false conditions, such as “Quantity”>16 AND “Quantity”<13). Such judgment conditions can never be triggered and need to be deleted with an error message.

[0105] 2. Bloodline tracing (see attached document for detailed steps) Figure 4 )

[0106] (1) Using field-level traceability, the monetary unit of the financial system is converted into the monetary unit of project management, such as "ten thousand yuan", and the conversion path of each field is recorded.

[0107] (2) Abnormal data link marker: Mark the data that has been converted abnormally. The red dashed arrow indicates the error path, and an error code is attached (e.g., ERR_CODE=0x8F: The amount field length in the financial system is 20 digits, while the amount field length in the project management system is 15 digits, and the data conversion is abnormal).

[0108] (3) Implement feedback optimization strategies.

[0109] (4) Rule layer reconfiguration: Based on the abnormal feedback and modifications of lineage tracing, recompile the AST structure tree.

[0110] 3. Intelligent Acquisition Layer

[0111] (1) Select the Flink engine or Spark engine based on the amount of data in the link.

[0112] (2) When the data volume is greater than 0.8TB and the delay is less than 5 minutes, the Flink engine is enabled and the checkpoint interval is set to 300 seconds.

[0113] (3) When batch processing data is selected, the Spark engine is enabled, and the number of partitions is fixed at 200. The database automatically distributes the data into 200 physical sub-tables based on the hash value. For frequently queried fields, local indexes are created to avoid performance degradation in queries.

[0114] (4) Start the data cleaning component library.

[0115] (5) Perform data validation on fields with strict requirements, such as amount and project fields, and generate alarm work orders based on the results.

[0116] (6) Standardize and convert data of the same type and with data specifications, such as dates, project categories, and fund items, for example, by using regular expressions to match dates in various formats to verify the accuracy of the data.

[0117] 4. Intelligent Conversion Engine Layer

[0118] (1) Generate code for the above configuration items according to the configuration requirements.

[0119] (2) Convert configuration items such as conditional branching, data transformation, and data mapping into SQL statements.

[0120] (3) Generate Java classes for complex operations such as logical judgments and execution plans.

[0121] In this embodiment, see the attached Figure 3 The specific process of smart field mapping is as follows:

[0122] 1. Input layer and preprocessing

[0123] (1) Receive source data fields and target structure fields.

[0124] (2) Standardization: unify naming format (all uppercase → camelCase), data type (string → numeric), (character length 20 → character length 32), etc.

[0125] 2. Multi-dimensional feature extraction

[0126] (1) Semantic feature extraction: Use the BERT model to analyze the contextual similarity of the fields.

[0127] (2) Structural feature matching: Compare field length, field type, field name, constraints and other conditions for matching.

[0128] (3) Naming pattern analysis: Calculate the edit distance of field names (e.g., the distance between "Amount" and "Amt" is 3).

[0129] 3. Similarity Calculation and Decision

[0130] (1) Weighted composite score: 0.6×semantic similarity + 0.3×structural matching degree + 0.1×naming similarity.

[0131] (2) If the total score is ≥0.92, the mapping will be automatic; otherwise, the manual intervention interface will be triggered to display the candidate fields of the top three.

[0132] 4. Feedback closed-loop optimization

[0133] (1) Users manually correct the mapping relationship (e.g., map “Proj_No” to “Project_ID”).

[0134] (2) The corrected results are stored in the knowledge base, the training set of the BERT model is updated, and the accuracy of subsequent matching is improved.

[0135] In this embodiment, see the attached Figure 4 The data lineage tracing process is as follows:

[0136] 1. Interface initialization and data loading

[0137] (1) Activate the data lineage tracking function

[0138] ① When a user clicks the “Data Lineage Tracing” function entry in the system interface, the system loads a preset data source (such as a financial system, a science and technology system, a project management platform, etc.).

[0139] ② Input parameter configuration: Users select the data range (full / incremental) and time interval.

[0140] (2) Automatic data source association

[0141] ① The system automatically parses the relationship between the data input layer and the data storage layer based on the data source type selected by the user.

[0142] ② Error Handling: If the data source connection fails, the interface will display a red warning icon and prompt "Connection error: Please check network or permission configuration".

[0143] 2. Data lineage chart display

[0144] (1) Layered visual display

[0145] ① Data Input Layer: Displays the original data source (such as "Financial System.Budget Table" in the database).

[0146] ② Data processing layer: Displays the conversion logic (e.g., "Unit conversion (ten thousand yuan → yuan)").

[0147] ③ Application output layer: Displays the final output (such as "Project Management. Fund Disbursement Instruction").

[0148] ④ External system interaction: Display external interfaces (such as "Audit System Alarm Interface").

[0149] (2) Link dynamic rendering

[0150] The system automatically draws arrow lines based on the data flow path:

[0151] Solid green line: Normal data flow (e.g., "Financial System → Conversion Engine → Project Management").

[0152] Red dashed line: Abnormal data link (such as "conversion engine → audit system" not being connected).

[0153] Node status indication:

[0154] Green indicator light: The node is operating normally.

[0155] Red indicator light: There is an anomaly in the node (such as data cleaning failure).

[0156] 3. Interactive Function Operation Flow

[0157] (1) Viewing node metadata

[0158] ① Clicking a node: When a user clicks the "Unit Conversion" node, a sidebar will pop up displaying the following information:

[0159] ② Conversion rules: Unit conversion logic ("Amount × 10000, amount converted to 'ten thousand yuan'").

[0160] ③ Association rule: The rule chain that triggers this conversion ("Project progress is greater than or equal to 80%").

[0161] ④ Execution Log: Records of the last 5 conversions, showing the timestamp, input value, and output value.

[0162] ⑤ Tracing the source of abnormal data links: Click on the red dotted link to display the reason for the abnormality (such as "negative value in the amount field", "system connection failure", etc.).

[0163] (2) Time range filtering

[0164] ① Time selector operation: Users can drag the timeline or enter start and end times (such as "2024-10-01 to 2024-10-31"), and the interface will dynamically refresh the lineage graph, displaying only the data flow path within that time period.

[0165] ② Version comparison: Users select two time points (such as "2024-09" and "2024-10"), and the system highlights the differences in the links (such as the newly added "Audit Rules" branch).

[0166] (3) Exception handling and feedback

[0167] ① Mark as abnormal: When the user clicks on the abnormal node (red indicator light) and selects "Mark as processed", the system updates the link status to green.

[0168] ② Work order generation: If an exception requires manual intervention, click the "Generate Work Order" button to automatically fill in the exception details and push it to the audit department.

[0169] (4) Exporting the data lineage report

[0170] Export Operation: When the user clicks the "Export" button and selects a format (PDF / PNG / Excel), the system generates a report containing the following content:

[0171] ① Complete data lineage chart.

[0172] ②Key node metadata (transformation rules, execution logs).

[0173] ③Statistical table of abnormal data links (quantity, type, processing status).

[0174] 4. Abnormal Data Link Handling Process

[0175] (1) Anomaly detection and alarm

[0176] ①Automatic detection: The system scans the data links in real time. If an anomaly is detected (such as a field mapping failure rate of 5% or more), a floating alarm box will pop up on the interface, indicating "10 abnormal data links detected".

[0177] ② Notification prompt: Work orders are automatically generated and sent to relevant business and technical departments through the system.

[0178] (2) Human intervention and correction

[0179] ① Correct mapping relationship: The user double-clicks the field node in the abnormal data link (such as "Project Code"), manually selects the target field (such as "Project Sequence Number"), and clicks "Confirm Mapping".

[0180] ② Feedback loop: The correction results are stored in the knowledge base to optimize the subsequent intelligent matching model.

[0181] (3) Rollback of historical versions

[0182] ①Version Management: When a user clicks the "Historical Versions" button and selects a rule version at a specific point in time (such as "2023-10-01 Rule V2"), the system restores the lineage chart for that version.

[0183] ② Difference Comparison: The interface highlights the differences between versions in blue (such as the newly added "Audit Rules" node).

[0184] In this embodiment, as can be seen from the above content and corresponding execution flow, the core content of the method of the present invention specifically includes:

[0185] 1. Multimodal Visual Rule Modeling Engine

[0186] (1) Dynamic bidirectional conversion mechanism

[0187] Visualization → Code Generation: Users build rule chains by dragging and dropping components (such as conditional statements, field mappings, and trigger actions), and the system generates executable code (such as Spark SQL and Java classes) in real time.

[0188] Code → Visual Reverse Engineering: Supports importing historical scripts and automatically restoring them into visual flowcharts, reducing the learning cost for business personnel.

[0189] (2) Semantic Enhancement Rule Component Library

[0190] Conditional components: support nested logic and have built-in semantic templates for the financial domain.

[0191] Mapping components: Integrates commonly used data standard mapping components, such as: unit conversion (ten thousand yuan ↔ yuan), format standardization (dates are unified as YYYY-MM-DD), encoding mapping (such as "male / female → M / F") and other high-frequency operations.

[0192] 2. Semantic intelligent parsing and cross-system alignment technology

[0193] Pre-training and fine-tuning: A proprietary training set is built based on 500,000+ fiscal terms (such as "budget item code" and "centralized treasury payment") to achieve field-level semantic alignment.

[0194] Multi-dimensional matching algorithm: Based on "semantic similarity" and "structural compatibility", it performs multi-dimensional intelligent matching, including field name, type, length, description, etc.

[0195] 3. End-to-end data lineage tracing

[0196] Field-level traceability graph: records the complete path of "financial system data → data cleaning → standard conversion → operation instructions", and supports backtracking of abnormal data links.

[0197] Closed-loop feedback mechanism: If the mapping error rate exceeds the threshold, an optimization signal is sent to the rule modeling layer to trigger AST reconstruction; If the data quality is abnormal (such as the null value rate exceeding the threshold), an adaptive resampling process (such as incremental verification + hash comparison) is initiated.

[0198] The results of comparing the technical effects of the method of the present invention with those of the traditional method are shown in Table 1.

[0199] Table 1 Comparison of Technical Effects

[0200]

[0201] Example 2

[0202] For personalized configurations for business personnel (S01 special fund allocation management scenario), the intelligent data collection and conversion method provided in Example 1 is used to implement the data collection and conversion process as follows:

[0203] 1. Rule configuration:

[0204] (1) Business personnel drag and drop components such as "Project Progress Verification" and "Fund Release Conditions".

[0205] (2) Setting logic: When the project progress is greater than or equal to 80% and the audit report is approved, connect the two through the "AND" logic component and bind it to the "fund disbursement" action component.

[0206] 2. Data Collection:

[0207] (1) Configure the database connection parameters of the fiscal system interface and the project management platform in the adapter, and automatically connect to the fiscal budget management system and the project management platform.

[0208] (2) Internal network data can be obtained and imported by passing through the network wall to obtain relevant files or original data files.

[0209] (3) Incremental acquisition of real-time data: Only data rows whose modification time is later than the last data synchronization time and whose MD5 hash value has changed are collected. Incremental data acquisition is performed by polling every 30 minutes.

[0210] 3. Execution conversion:

[0211] The engine automatically generates Spark SQL and submits it to the cluster. The SQL content must be based on the rule engine configuration requirements. First, it queries the dataset containing project codes, project names, project progress, etc., whose data modification time is greater than the last data synchronization time. At the same time, it filters out data whose project progress is greater than or equal to 80%, synchronizes this batch of data, and automatically executes the relevant action components for fund disbursement.

[0212] 4. Result Output:

[0213] (1) Generate a fund disbursement instruction and push it to the relevant execution department. After the execution department reviews it, it clicks the "Execute Fund Disbursement" button. The system executes the instruction and transfers data according to the configuration requirements of the relevant action components for fund disbursement.

[0214] (2) Output a visual monitoring dashboard (including a fund-project relationship view)

[0215] Example 3

[0216] For automatically generated scenarios (management of suspected unreasonable expenditures in project funding), the intelligent data collection and conversion method provided in Example 1 is used to implement the data collection and conversion process as follows:

[0217] 1. Automatic rule generation:

[0218] (1) Based on the various management requirements for expenditure, after deep learning and analysis, rules are automatically generated for a certain project expenditure that is suspected to be unreasonable.

[0219] (2) Generation logic: When the business expenses include funds for publication, literature and other purposes, and the similar projects are identified as being in violation, the subsequent processing of fund supervision is triggered.

[0220] 2. Data Collection:

[0221] (1) Automatically connect to the fiscal budget management system and project management platform

[0222] (2) Internal network data can be obtained and imported by passing through the network wall to obtain relevant files or original data files.

[0223] (3) Incremental acquisition of real-time data: Only data rows whose modification time is later than the last data synchronization time and whose MD5 hash value has changed are collected. Incremental data acquisition is performed by polling every 30 minutes.

[0224] 3. Execution conversion:

[0225] The engine automatically generates Java or Python code blocks. The code content must adhere to the intelligent conversion engine's configuration requirements and invoke fund supervision triggers. First, it checks whether the fund usage contains sensitive keywords (such as 'publication' or 'document'). If no violations are found, the supervision rules are not triggered. If violations are found, the severity is assessed, a violation work order is generated, and it is distributed to relevant departments for rectification.

[0226] 4. Result Output:

[0227] (1) Triggering the violation conditions, the violation content is pushed to relevant personnel via message.

[0228] (2) Output a visual monitoring dashboard (including a fund-project related view), information on violations, and relevant management regulations.

[0229] (3) Follow up on the subsequent rectification and implement other management requirements.

[0230] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0231] This invention provides a data intelligent collection and transformation method based on a visual rule model, featuring a two-way visualization → code conversion mechanism: It pioneers a "what you see is what you get" rule designer, supporting business personnel or technical support personnel to configure complex logic without coding, while retaining the code-level flexibility of developers. A semantic enhancement model for the financial domain: Based on a domain-pre-trained BERT intelligent mapping algorithm, it solves similar semantic gap problems in traditional solutions. A dynamic policy engine and elastic computing: Combining rule version management (canary release / rollback) and distributed engine self-adaptation, it achieves a minute-level closed loop of "policy change → rule adjustment → execution taking effect".

[0232] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data intelligent acquisition and transformation method based on a visual rule model, characterized in that: Includes the following steps, S1. The visual rule modeling layer converts the configured rule chain into a JSON format data body, parses the data body into an AST (Abstract Syntax Tree), and the intelligent field mapping module of the visual rule modeling layer performs field mapping on the AST. The field mapping process is as follows: receiving source data fields and target structure fields and standardizing them; using a BERT model pre-trained in the financial domain to analyze the contextual semantic similarity between the source data fields and the target structure fields, performing structural feature matching, calculating the edit distance of field names, and calculating a weighted comprehensive score based on "0.6 × semantic similarity + 0.3 × structural matching + 0.1 × naming similarity". If the score is ≥0.92, automatic mapping and verification are performed; otherwise, manual intervention is triggered. Then, the intelligent conversion engine is used to convert the mapped fields into executable code and generate an execution plan. S2. The intelligent acquisition layer pulls data from heterogeneous data sources based on full acquisition and / or incremental acquisition. The intelligent acquisition layer adaptively selects the engine according to the data volume and latency requirements: when the data volume is greater than 0.8TB and the latency requirement is less than 5 minutes, the Flink engine is enabled; when batch processing data, the Spark engine is enabled. After performing dual verification of timestamp and content hash and standardization cleaning on the extracted data, it is partitioned by primary key hash and allocated to the intelligent transformation engine for parallel processing. The S3 intelligent conversion engine's conversion process visualization module monitors the data lineage graph in real time, records field-level conversion paths, and sends signals to the visualization rule modeling layer or intelligent acquisition layer based on the type of anomaly to achieve rule chain reconfiguration or data re-acquisition. Specifically, if the field mapping error rate is detected to be greater than a preset threshold, an optimization signal is sent to the visualization rule modeling layer to trigger rule chain reconfiguration; if data quality anomalies are detected, an alarm is sent to the intelligent acquisition layer to initiate the data re-acquisition process, thus achieving a closed loop of "monitoring-anomaly identification-feedback adjustment".

2. The data intelligent acquisition and transformation method based on a visual rule model according to claim 1, characterized in that: Step S1 specifically includes the following: S11. Users configure rule chains in the drag-and-drop rule component library of the visual rule modeling layer; S12, the dynamic semantic parsing engine of the visual rule modeling layer converts the rule chain into an intermediate representation in JSON format, which includes component types, parameters and connection relationships; S13, the dynamic semantic parsing engine of the visualization rule modeling layer further parses the intermediate representation into an AST abstract syntax tree and optimizes and merges redundant nodes; S14. The intelligent field mapping module of the visualization rule modeling layer performs field mapping on the AST abstract syntax tree and verifies the mapping error rate. S15. The mapped fields are converted into executable code through the dynamic rule compilation module of the intelligent transformation engine, and Spark SQL or Java classes are generated in real time according to the scenario. S16. Submit the executable code to the distributed execution framework of the intelligent transformation engine to generate a physical execution plan.

3. The data intelligent acquisition and transformation method based on a visual rule model according to claim 2, characterized in that: In step S14, the field mapping process of the intelligent field mapping module is as follows: S141. Data Input and Preprocessing: Receive source data fields and target structure fields, and standardize them. S142. Multi-dimensional feature extraction: Use the BERT model to analyze the contextual similarity between the source data fields and the target structure fields, perform structural feature matching between the source data fields and the target structure fields, and calculate the edit distance of the field names between the source data fields and the target structure fields; S143. Similarity Calculation and Decision: Calculate the weighted comprehensive score between the source data field and each target structure field based on the semantic similarity, structural matching degree and naming similarity between the source data field and the target structure field. When the weighted comprehensive score is greater than or equal to the total score threshold, the source data field is automatically mapped to the corresponding target structure field, and the mapping result is verified. If the verification passes, a mapping scheme is generated and a mapping relationship table is output. If the weighted composite score is less than the total score threshold or the mapping result fails the verification, the manual intervention interface will be triggered to display the top three candidate target structure fields. S144, Feedback closed-loop optimization: Users manually correct the mapping relationship and store the correction results in the knowledge base to update the training set of the BERT model and improve the subsequent matching accuracy.

4. The data intelligent acquisition and transformation method based on a visual rule model according to claim 1, characterized in that: Step S2 specifically includes the following: S21. The multi-source adapter of the intelligent acquisition layer pulls data from heterogeneous data sources; S22. For incremental data collection, the incremental data collection strategy module of the intelligent data collection layer will perform dual verification of the pulled data using timestamp and content hash. S23, The data cleaning module of the intelligent acquisition layer performs standardization processing on the pulled data; S24. The standardized data is partitioned according to the primary key hash and distributed to the cluster nodes of the distributed execution framework for parallel processing.

5. The data intelligent acquisition and transformation method based on a visual rule model according to claim 4, characterized in that: The intelligent acquisition layer selects either the Flink engine or the Spark engine to pull data from heterogeneous data sources based on the data volume of the link. When the data volume is greater than 0.8TB and the latency is less than 5 minutes, the Flink engine is enabled, and the checkpoint interval is set to 300 seconds. When batch processing of data is selected, the Spark engine is enabled, the number of partitions is fixed at 200, and the database automatically distributes the data into 200 physical sub-tables based on the hash value. For frequently queried fields, local indexes are created to avoid performance degradation in queries.

6. The data intelligent acquisition and transformation method based on a visual rule model according to claim 1, characterized in that: Step S3 specifically includes the following: S31, the intelligent conversion engine's conversion process visualization module monitors the data lineage graph in real time and records the field-level conversion path; S32. If the field mapping error rate is greater than the preset threshold, an optimization signal is sent to the visualization rule modeling layer to trigger the reconfiguration of the rule chain. S33. If the data quality is abnormal, an alarm will be sent to the intelligent acquisition layer to start the data re-acquisition process.

7. The data intelligent acquisition and transformation method based on a visual rule model according to claim 6, characterized in that: The monitoring process and results of the data pedigree mapping are displayed using the data pedigree tracing interface. The data lineage tracing interface includes a data input layer for displaying the original data source, a data processing layer for displaying the conversion logic, an application output layer for displaying the final output, and an external system interaction layer for displaying external interfaces. The data lineage tracking interface can dynamically render the data link and automatically draw arrow connections based on the data flow path. Normal and abnormal data links are represented by different colors and linear connection lines, respectively. At the same time, the interface is equipped with node indicator lights, which are used to represent normally operating nodes and abnormal nodes.

8. The data intelligent acquisition and transformation method based on a visual rule model according to claim 7, characterized in that: The interactive operation flow of the data lineage tracking interface is as follows: A1. Node Metadata Viewing: By clicking on a node, you can view the transformation rules, association rules, execution logs, and abnormal data links; for abnormal data links, clicking on the connecting line representing the link will show the reason for the data link abnormality; A2. Time Range Filtering: By dragging the timeline or entering the start and end times, the interface dynamically refreshes the data lineage graph, displaying only the data flow path for the corresponding time period; by selecting two different time periods, the difference links can be highlighted. A3. Anomaly Handling and Feedback: By clicking on an anomaly node, it can be marked as processed and the corresponding link status can be updated; if manual intervention is required, the anomaly details will be automatically filled in to generate an anomaly work order, which will be pushed to the audit department. A4. Exporting Data Lineage Report: By clicking the export button and selecting the export format, you can generate and export a data lineage report.

9. The data intelligent acquisition and transformation method based on a visual rule model according to claim 8, characterized in that: The exported data lineage report includes a complete data lineage map, metadata of key nodes, and a statistical table of abnormal data links.

10. The data intelligent acquisition and transformation method based on a visual rule model according to claim 7, characterized in that: The handling process for abnormal data links in the data lineage tracking interface is as follows: B1. Anomaly Detection and Alarm: Real-time scanning of data links. If an anomaly is detected, a floating alarm box will pop up on the data lineage tracing interface, indicating the number of abnormal data links detected. An abnormal data link work order will be automatically generated and sent to the relevant business and technical departments. B2. Manual Intervention and Correction: Click on the field node in the abnormal data link, manually select the target field, and click Confirm Mapping to correct the mapping relationship; The corrected results are stored in a knowledge base to optimize subsequent intelligent matching models; B3. Historical Version Rollback: Click the Historical Version button, select the rule version at a certain point in time, and the data lineage graph of that version will be automatically restored; the data lineage tracking interface uses color highlighting to show the differences between versions.

Images