Intelligent data acquisition and conversion method based on visual rule model

Abstract

The invention discloses an intelligent data acquisition and conversion method based on a visual rule model, which comprises the following steps: S1, converting a configured rule chain into a data body in a JSON format by a visual rule modeling layer, analyzing the data body into an AST abstract syntax tree, and performing field mapping on the AST abstract syntax tree, the mapping fields are converted into executable codes through an intelligent conversion engine, and an execution plan is generated; s2, the intelligent acquisition layer pulls data from the heterogeneous data source, cleans the data and then allocates the data to an intelligent conversion engine for parallel processing; and S3, the intelligent conversion engine monitors the data consanguinity map and the field-level conversion path in real time, and sends a signal to the visual rule modeling layer or the intelligent acquisition layer according to the abnormal condition type so as to realize rule chain reconfiguration or data reacquisition. The method has the advantages that non-coding docking and real-time supervision of heterogeneous system data are realized, and the problems of dynamic adaptation, semantic understanding and data blood relationship interpretability in financial fund management are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data intelligent processing, and particularly relates to a data intelligent acquisition and conversion method based on a visualization rule model. Background Art

[0002] With the rapid development of the social economy, the scale of fiscal funds continues to expand, and the flow of funds involves multi-level government departments, enterprises and institutions, and the public service field. However, the cross-departmental collaborative management of fiscal funds faces severe challenges, specifically manifested as follows: 1. Problem of data islands in heterogeneous systems Data structure conflict: The finance department adopts the Golden Finance Project (based on the Oracle database, with field naming in camel case), while the project management system adopts a collaborative platform (based on databases such as MySQL and Highgo, with field naming in underscore case), and due to the naming rule differences, system automatic matching cannot be achieved.

[0003] Business logic inconsistency: The definition methods of fund appropriation rules are different in different systems (for example, "project progress ≥ 80%" is a numeric field in the Golden Finance Project, while it is a percentage string in the project management system).

[0004] 2. Low efficiency of manual processing Time-consuming for cross-system docking: Dependence on Excel or scripts for field mapping and format conversion, and a single data docking takes 3 - 5 working days, and manual operations are prone to errors (such as omitting the multiplier factor when converting the unit "ten thousand yuan → yuan", resulting in common sense problems such as amount errors).

[0005] Lag in rule adjustment: For policy changes (such as the fund appropriation condition being adjusted from "progress ≥ 80%" to "progress ≥ 70% and audit rating ≥ B"), it is necessary to re-develop SQL / Python / java scripts, and the average response cycle exceeds 2 weeks.

[0006] 3. Insufficient technical threshold and dynamic adaptation ability Limitations of ETL tools: Traditional tools require writing XML configuration files, and business personnel cannot directly participate in rule configuration through a visualization interface.

[0007] Lack of semantic understanding: Existing solutions rely on regular expression matching of field names or manual matching, and cannot recognize business semantics (such as the equivalence relationship between "Proj_No" and "Project_ID"), resulting in a high mapping error rate and long time consumption.

[0008] 4. Defects in data lineage tracking and interpretability Fragmented links: Existing technologies only record the flow of data at the table level and lack field-level conversion path tracking. For example, it's impossible to trace the complete processing chain of the "amount" field from raw data to fund disbursement instructions (including cleaning, unit conversion, and rule triggering).

[0009] However, traditional cross-departmental data integration methods (such as traditional ETL tools, API customized docking solutions, data middle-office architecture, and open source rule engines) are unable to solve the above problems due to their own shortcomings.

[0010] (1) Disadvantages of traditional ETL tools (such as Informatica / Kettle): Lack of visual rule modeling capabilities requires developers to write complex configuration files.

[0011] Dynamic semantic parsing is not supported, and nested conditional judgments must be hard-coded (such as multi-layer IF-ELSE logic).

[0012] (2) Disadvantages of customized API docking solutions: The development cost of a single interface is as high as 10 man-days, and multi-version compatibility needs to be maintained (for example, upgrading the financial system API causes historical interfaces to become invalid).

[0013] Incremental data synchronization relies on a polling mechanism, which has poor real-time performance (delay ≥ 1 hour) and cannot meet the minute-level response requirements for fund disbursement.

[0014] (3) Disadvantages of the data center architecture (such as Alibaba DataWorks): The implementation period is long (≥3 months), and small and medium-sized institutions find it difficult to afford the resource investment.

[0015] The rule engine is highly coupled with the business system, and dynamic adjustments require downtime for deployment, affecting business continuity.

[0016] (4) Disadvantages of open source rule engines (such as Drools): Rule logic must be defined in code (such as DRL language) and cannot be edited directly by business personnel.

[0017] The lack of intelligent field mapping and closed-loop feedback on data lineage results in low anomaly location efficiency (average troubleshooting time ≥ 4 hours). Summary of the Invention

[0018] The purpose of the present invention is to provide a data intelligent collection and conversion method based on a visual rule model, thereby solving the above-mentioned problems existing in the prior art.

[0019] In order to achieve the above object, the technical solution adopted by the present invention is as follows: A data intelligent acquisition and conversion method based on a visualization rule model, comprising the following steps: S1. The visualization rule modeling layer converts the configured rule chain into a data body in JSON format, parses the data body into an AST abstract syntax tree, performs field mapping on the AST abstract syntax tree, and then uses an intelligent conversion engine to convert the mapped fields into executable code to generate an execution plan; S2. The intelligent acquisition layer pulls data from heterogeneous data sources based on full-volume acquisition and / or incremental acquisition, cleans the data, and assigns it to the intelligent conversion engine for parallel processing; S3. The intelligent conversion engine monitors the data lineage graph in real time, monitors the field-level conversion path, and sends signals to the visualization rule modeling layer or the intelligent acquisition layer according to the type of exception situation to reconfigure the rule chain or re-acquire data.

[0020] Preferably, step S1 specifically includes the following content: S11. The user configures a rule chain in the drag-and-drop rule component library of the visualization rule modeling layer; S12. The dynamic semantic parsing engine of the visualization rule modeling layer converts the rule chain into an intermediate representation in JSON format, including component types, parameters, and connection relationships; S13. The dynamic semantic parsing engine of the visualization rule modeling layer further parses the intermediate representation into an AST abstract syntax tree, optimizing and merging redundant nodes; S14. The intelligent field mapping module of the visualization rule modeling layer performs field mapping on the AST abstract syntax tree and verifies the mapping error rate; S15. The mapped fields are converted into executable code through the dynamic rule compilation module of the intelligent conversion engine, and Spark SQL or Java classes are generated in real time according to the scenario; S16. The executable code is submitted to the distributed execution framework of the intelligent conversion engine to generate a physical execution plan.

[0021] Preferably, in step S14, the field mapping process of the intelligent field mapping module is as follows: S141. Data input and preprocessing: Receive source data fields and target structure fields and perform standardization processing on them; S142. Multi-dimensional feature extraction: Use the BERT model to analyze the context similarity between the source data field and the target structure field description, perform structural feature matching on the source data field and the target structure field, and calculate the field name edit distance between the source data field and the target structure field; S143. Similarity Calculation and Decision: Calculate the weighted comprehensive score between the source data field and each target structure field based on the semantic similarity, structure matching degree, and naming similarity between the source data field and the target structure field. When the weighted comprehensive score is greater than or equal to the total score threshold, automatically map the source data field to the corresponding target structure field, and verify the mapping result. If the verification passes, generate a mapping scheme and output a mapping relationship table; if the weighted comprehensive score is less than the total score threshold or the mapping result verification fails, trigger an artificial intervention interface to display the top three candidate target structure fields. S144. Feedback Closed-loop Optimization: The user manually corrects the mapping relationship, stores the correction result in the knowledge base, updates the training set of the BERT model, and improves the subsequent matching accuracy.

[0022] Preferably, step S2 specifically includes the following content. S21. The multi-source adapter in the intelligent acquisition layer pulls data from heterogeneous data sources. S22. For incremental acquisition, the incremental acquisition strategy module in the intelligent acquisition layer performs double verification of the pulled data by timestamp and content hash. S23. The data cleaning module in the intelligent acquisition layer standardizes the pulled data. S24. The standardized data is partitioned according to the primary key hash and distributed to the cluster nodes of the distributed execution framework for parallel processing.

[0023] Preferably, the intelligent acquisition layer selects the Flink engine or the Spark engine to pull data from heterogeneous data sources according to the data volume of the link. When the data volume is greater than 0.8TB and the latency is less than 5 minutes, the Flink engine is enabled, and the checkpoint interval is set to 300 seconds; when batch processing data is selected, the Spark engine is enabled, the number of partitions is fixed at 200, and the database automatically distributes the data to 200 physical sub-tables according to the hash value. For frequently queried fields, local indexes are established to avoid a decline in query performance.

[0024] Preferably, step S3 specifically includes the following content. S31. The conversion process visualization module of the intelligent conversion engine monitors the data lineage graph in real time and records the field-level conversion path. S32. If the field mapping error rate is greater than the preset threshold, send an optimization signal to the visualization rule modeling layer to trigger the reconfiguration of the rule chain. S33. If the data quality is abnormal, send an alarm to the intelligent acquisition layer and start the data re-acquisition process.

[0025] Preferably, the process and results of data lineage graph monitoring are displayed using the data lineage tracking interface. The data lineage tracking interface includes a data input layer for displaying the source of the original data, a data processing layer for displaying the conversion logic, an application output layer for displaying the final output, and an external system interaction layer for displaying external interfaces; The data lineage tracking interface can achieve dynamic rendering of the link. According to the data flow path, arrows are automatically drawn to connect. Different colors and line styles of connection lines are used to represent normal data links and abnormal data links respectively. At the same time, node indicator lights are set in the interface, and different colors of indicator lights are used to represent normal-running nodes and nodes with anomalies.

[0026] Preferably, the interaction operation process of the data lineage tracking interface is as follows: A1. Node metadata viewing: By clicking on a node, the conversion rule, association rule, execution log, and abnormal data link can be viewed. For an abnormal data link, by clicking on the connection line representing the link, the reason for the data link anomaly can be displayed; A2. Time range filtering: By dragging the time axis or entering the start and end times, the interface dynamically refreshes the data lineage graph, only displaying the data flow paths for the corresponding time period. By selecting two different time periods, the differential links can be highlighted; A3. Abnormal handling and feedback: By clicking on an abnormal node, it can be marked as processed, and the corresponding link status is updated. If manual intervention is required, the abnormal details are automatically filled to generate an abnormal work order and pushed to the audit department; A4. Data lineage report export: By clicking the export button and selecting the export format, a data lineage report can be generated and exported.

[0027] Preferably, the exported data lineage report includes a full data lineage map, key node metadata, and a statistical table of abnormal data links.

[0028] Preferably, the processing process of abnormal data links in the data lineage tracking interface is as follows: B1. Abnormal detection and warning: The data link is scanned in real time. If an anomaly is detected, a floating warning box pops up on the data lineage tracking interface, indicating the number of detected abnormal data links. An abnormal data link work order is automatically generated and sent to the relevant business departments and technical departments; B2. Manual intervention and correction: By clicking on the field node in the abnormal data link, manually select the target field, and click to confirm the mapping to correct the mapping relationship. The correction result is stored in the knowledge base for optimizing the subsequent intelligent matching model; B3. Historical version rollback: By clicking the historical version button and selecting the rule version at a certain time point, the data lineage graph of that version is automatically restored. The data lineage tracking interface uses color highlighting to show the differences between versions.

[0029] The beneficial effects of the present invention are as follows: 1. Two-way visualization → code conversion mechanism: The "what you see is what you get" rule designer is pioneered, supporting non-coding configuration of complex logics by business personnel or technical support personnel, while retaining the code-level flexibility for developers. 2. Semantic enhancement model in the financial field: An intelligent mapping algorithm based on domain pre-trained BERT is used to solve the problem of similar semantic gaps in traditional solutions. 3. Dynamic policy engine and elastic computing: Combining version management of rules (gray release / rollback) with self-adaptation of the distributed engine, a minute-level closed loop of "policy change → rule adjustment → execution and effectiveness" is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is the system architecture diagram of the acquisition and conversion method in the embodiment of the present invention; Figure 2 is the flow chart of visual rule configuration in the embodiment of the present invention; Figure 3 is the schematic diagram of the intelligent field mapping algorithm in the embodiment of the present invention; Figure 4 is the schematic diagram of the data lineage tracking interface in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Embodiment

[0032] In this embodiment, a data intelligent acquisition and conversion method based on a visual rule model is provided. Different from traditional cross-department data fusion methods, this method realizes code-free docking and real-time supervision of heterogeneous system data through a technical framework of "visual rule modeling + semantic intelligent parsing + dynamic policy engine". This method abandons the "hard coding" mode of traditional solutions and solves the problems of dynamic adaptation, semantic understanding and data lineage interpretability in financial fund management based on the following core technical breakthroughs.

[0033] As Figure 1 shown, it can be seen from the combined system architecture that the data intelligent acquisition and conversion method mainly includes three main interaction processes. I. Interaction between the visual rule modeling layer and the intelligent conversion engine The visual rule modeling layer converts the configured rule chain into a data body in JSON format, parses the data body into an AST abstract syntax tree, performs field mapping on the AST abstract syntax tree, and then uses the intelligent conversion engine to convert the mapped fields into executable code to generate an execution plan. The specific contents are as follows. 1.1. Users configure rule chains (such as "Project progress ≥ 80% and audit passed → Trigger fund appropriation") in the drag-and-drop canvas of the visual rule modeling layer.

[0034] 1.2. The dynamic semantic parsing engine converts the rule chain into an intermediate representation in JSON format, including component types, parameters, and connection relationships.

[0035] 1.3. The intermediate representation is further parsed into an abstract syntax tree (AST), and redundant nodes (such as duplicate field mappings) are optimized and merged.

[0036] 1.4. The AST is converted into executable code through the dynamic rule compilation module, generating Spark SQL or Java classes according to the scenario.

[0037] 1.5. The executable code is submitted to the distributed execution framework (Spark / Flink) to generate a physical execution plan (such as a Spark DAG).

[0038] II. Interaction between the intelligent collection layer and the intelligent conversion engine The intelligent collection layer pulls data from heterogeneous data sources based on full-volume collection and / or incremental collection methods, and after cleaning the data, distributes it to the intelligent conversion engine for parallel processing. The specific contents are as follows. 2.1. The multi-source adapter pulls data from heterogeneous data sources (such as Kingdee financial system API, Oracle database).

[0039] Full-volume collection: Execute SELECT * FROM table for the first connection.

[0040] Incremental collection: Based on double verification of timestamp and content hash.

[0041] 2.2. The data cleaning module performs standardization processing on the collected data: Null value filling: Numerical fields are filled with 0, and character fields are filled with "N / A".

[0042] Anomaly detection: Detect negative amounts and trigger an alarm.

[0043] 2.3. The cleaned data is partitioned by primary key hash and distributed to cluster nodes for parallel processing.

[0044] III. Interaction between the intelligent conversion engine and the visual rule modeling layer as well as the intelligent collection layer The intelligent conversion engine monitors the data lineage graph in real time, monitors the field-level conversion path, and sends signals to the visual rule modeling layer or the intelligent collection layer according to the type of abnormal situation to achieve reconfiguration of the rule chain or data recollection. The specific contents are as follows. 3.1. The conversion process visualization layer monitors the data lineage graph in real time and records the field-level conversion path (such as "Financial System.Amount → Conversion Engine.Unit Conversion → Project Management.Amount").

[0045] 3.2. If the field mapping error rate > 5%, send an optimization signal to the rule modeling layer to trigger the reconfiguration of the rule chain.

[0046] 3.3. If the data quality is abnormal (such as the null value rate exceeds the threshold), send an alarm to the acquisition layer and start the data re-acquisition process.

[0047] In this embodiment, see the appendix Figure 2 , the specific process of visual rule configuration and execution is as follows: 1. Drag and drop components (1). The user drags and drops behavior components, including condition, mapping, trigger type and other components. (2). The system generates a data body in JSON format, including component node IDs, parameters, connection relationships, etc. Generate the following format: {"node_id":"cond_001","type":"threshold","expr":"progress>=0.8"} (3). Perform structure optimization during AST generation.

[0048] (4). Merge duplicate mappings. Identify the same field conversion operations through SHA-256 hash comparison. For example, if the measurement unit of this field is "ten thousand yuan" multiple times, and the measurement unit this time is "yuan", the system adjusts the AST structure and gives a prompt message.

[0049] (5). Eliminate redundant branches and delete always-false conditions, such as "quantity" > 16 AND "quantity" < 13). Such judgment conditions can never be triggered and need to be deleted and an error message is prompted.

[0050] 2. Lineage tracing (see the appendix for detailed steps Figure 4 ) (1). Use field-level tracing to convert the measurement unit of the amount in the financial system to the measurement unit of the amount in project management, such as converting to "ten thousand yuan" in such scenarios, and record the conversion path of each field.

[0051] (2). Mark abnormal data links. Mark the data with abnormal conversions. Red dotted arrows indicate error paths, accompanied by error codes (such as ERR_CODE = 0x8F: The length of the amount field in the financial system is 20 digits, and the length of the amount in the project management system is 15 digits, and the data conversion is abnormal.). (3). Execute the feedback optimization strategy.

[0052] (4) Reconfigure the rule layer, and recompile the AST structure tree according to the abnormal feedback and modification of blood relationship tracking.

[0053] 3. Intelligent acquisition layer (1) Select the Flink engine or Spark engine according to the data volume of the link.

[0054] (2) When the data volume is greater than 0.8TB and the latency is less than 5 minutes, enable the Flink engine, and set the checkpoint interval to 300 seconds.

[0055] (3) When selecting to process data in batches, enable the Spark engine, fix the number of partitions at 200, and the database automatically distributes the data to 200 physical sub-tables according to the hash value. For frequently queried fields, establish local indexes to avoid degradation of query performance.

[0056] (4) Start the data cleaning component library.

[0057] (5) Perform data verification on content with strict requirements such as amount fields and project fields, and generate alarm work orders for the execution results.

[0058] (6) Perform standardized configuration and conversion on data of the same type with data specifications such as dates, project classifications, and fund subjects, such as using regular expressions to match multiple date formats to verify the accuracy of the data.

[0059] 4. Intelligent conversion engine layer (1) Generate code according to the above configuration items according to the configuration requirements.

[0060] (2) Convert configuration items such as conditional branches, data conversion, and data mapping into SQL statements.

[0061] (3) Generate java classes for complex operations such as logical judgment and execution plan.

[0062] In this embodiment, see Appendix Figure 3 , the specific process of intelligent field mapping is as follows: 1. Input layer and preprocessing (1) Receive source data fields and target structure fields.

[0063] (2) Standardize processing: unify the naming format (all uppercase → camel case), data type (string → numeric type), (character length 20 → character length 32), etc.

[0064] 2. Multi-dimensional feature extraction (1) Semantic feature extraction: Use the BERT model to analyze the context similarity of field descriptions.

[0065] (2) Structural feature matching: Match by comparing conditions such as field length, field type, field naming, and constraint conditions.

[0066] (3) Naming pattern analysis: Calculate the edit distance of field names (e.g., the distance between "Amount" and "Amt" is 3).

[0067] 3. Similarity calculation and decision-making (1) Weighted comprehensive score: 0.6 × semantic similarity + 0.3 × structural matching degree + 0.1 × naming similarity.

[0068] (2) If the total score ≥ 0.92, automatic mapping; otherwise, trigger the manual intervention interface to display the top three candidate fields.

[0069] 4. Feedback closed-loop optimization (1) The user manually corrects the mapping relationship (e.g., maps "Proj_No" to "Project_ID").

[0070] (2) The corrected result is stored in the knowledge base, and the training set of the BERT model is updated to improve the subsequent matching accuracy.

[0071] In this embodiment, see the appendix Figure 4 , and the data lineage tracing process is as follows: 1. Interface initialization and data loading (1) Start the data lineage tracing function ① The user clicks the "Data Lineage Tracing" function entry in the system interface, and the system loads the preset data sources (such as the financial system, science and technology system, project management platform, etc.).

[0072] ② Input parameter configuration: The user selects the data range (full volume / increment) and time interval.

[0073] (2) Automatic association of data sources ① The system automatically analyzes the association relationship between the data input layer and the data storage layer according to the data source type selected by the user.

[0074] ② Exception handling: If the data source connection fails, a red warning icon is displayed on the interface and a prompt "Connection exception: Please check the network or permission configuration" is given.

[0075] 2. Data lineage graph display (1) Hierarchical visualization display ① Data input layer: Display the original data source (e.g., "Financial System.Budget Table" in the database).

[0076] ② Data processing layer: Display the conversion logic (e.g., "Unit conversion (ten thousand yuan → yuan)").

[0077] ③ Application Output Layer: Displays the final output (such as "Project Management. Fund Allocation Instruction").

[0078] ④ External System Interaction: Displays external interfaces (such as "Audit System Alarm Interface").

[0079] (2) Link Dynamic Rendering The system automatically draws arrow connections according to the data flow path: Solid green line: Normal data link (such as "Financial System → Conversion Engine → Project Management").

[0080] Dashed red line: Abnormal data link (such as "Conversion Engine → Audit System" not being connected, etc.).

[0081] Node Status Indication: Green indicator light: The node is running normally.

[0082] Red indicator light: There is an abnormality in the node (such as data cleaning failure).

[0083] 3. Interactive Function Operation Process (1) Node Metadata Viewing ① Click on the node: The user clicks on the "Unit Conversion" node, and a sidebar pops up to display the following information: ② Conversion Rule: The unit conversion logic ("Amount × 10000, convert the amount to 'ten thousand yuan'").

[0084] ③ Association Rule: The rule chain that triggers this conversion ("Project progress is greater than or equal to eighty percent").

[0085] ④ Execution Log: The last 5 conversion records, showing the timestamp, input value, and output value.

[0086] ⑤ Abnormal Data Link Tracing: Click on the dashed red link to display the reason for the abnormality (such as "The amount field has negative values", "The system connection is not available", etc.).

[0087] (2) Time Range Filtering ① Time Selector Operation: The user drags the timeline or enters the start and end times (such as "2024-10-01 to 2024-10-31"), and the interface dynamically refreshes the lineage graph, only showing the data flow path within this time period.

[0088] ② Version Comparison: The user selects two time points (such as "2024-09" and "2024-10"), and the system highlights the different links (such as the newly added "Audit Rule" branch).

[0089] (3) Abnormality Handling and Feedback ① Marking anomalies: The user clicks on the anomaly node (red indicator light) and selects "Mark as processed", and the system updates the link status to green.

[0090] ② Work order generation: If manual intervention is required for the anomaly, click the "Generate Work Order" button, and the anomaly details will be automatically filled in and pushed to the audit department.

[0091] (4), Export of data lineage report Export operation: The user clicks the "Export" button, selects the format (PDF / PNG / Excel), and the system generates a report containing the following content: ① Complete data lineage diagram.

[0092] ② Metadata of key nodes (conversion rules, execution logs).

[0093] ③ Statistical table of abnormal data links (quantity, type, processing status).

[0094] 4. Processing flow of abnormal data links (1) Anomaly detection and alerting ① Automatic detection: The system scans the data link in real time. If an anomaly is detected (such as the field mapping failure rate is greater than or equal to five percent), a floating alert box will pop up on the interface, prompting "10 abnormal data links detected".

[0095] ② Notification prompt: A work order is automatically generated and sent to the relevant business departments and technical departments.

[0096] (2) Manual intervention and correction ① Correct mapping relationship: The user double-clicks on the field node in the abnormal data link (such as "project code"), manually selects the target field (such as "project serial number"), and clicks "Confirm mapping".

[0097] ② Feedback closed-loop: The correction result is stored in the knowledge base to optimize the subsequent intelligent matching model.

[0098] (3) Rollback of historical versions ① Version management: The user clicks the "Historical Versions" button and selects a rule version at a certain point in time (such as "Rule V2 on October 1, 2023"), and the system restores the data lineage graph of that version.

[0099] ② Difference comparison: The interface highlights the differences between versions in blue (such as the newly added "audit rule" node).

[0100] In this embodiment, it can be seen from the above content and the corresponding execution process that the core content of the method of the present invention specifically includes 1. Multimodal visualization rule modeling engine (1) Dynamic two-way conversion mechanism Visualization → Code Generation: Users construct a rule chain through drag-and-drop components (such as conditional judgment, field mapping, trigger actions), and the system generates executable code in real time (such as Spark SQL, Java classes).

[0101] Code → Visualization Reverse Parsing: Support importing historical scripts and automatically restoring them to visual flowcharts, reducing the learning cost for business personnel.

[0102] (2)Semantic Enhancement Rule Component Library Condition Components: Support nested logic and build semantic templates in the financial domain.

[0103] Mapping Components: Integrate common data standard mapping components, such as: unit conversion (ten thousand yuan ↔ yuan), format standardization (date unified to YYYY-MM-DD), encoding mapping (such as "male / female → M / F"), etc., for high-frequency operations.

[0104] 2. Semantic Intelligent Parsing and Cross-System Alignment Technology Pre-training and Fine-tuning: Build a proprietary training set based on more than 500,000 financial terms (such as "budget subject coding", "treasury centralized payment") to achieve field-level semantic alignment.

[0105] Multi-dimensional Matching Algorithm: Perform multi-dimensional intelligent matching based on "semantic similarity" and "structural compatibility", including field name, type, length, description, etc.

[0106] 3. Full-link Data Lineage Tracing Field-level Traceability Map: Record the complete path of "financial system data → data cleaning → standard conversion → operation instructions", and support backtracking of abnormal data links.

[0107] Closed-loop Feedback Mechanism: If the mapping error rate exceeds the threshold, send an optimization signal to the rule modeling layer to trigger AST reconstruction; if the data quality is abnormal (such as the null value rate exceeds the threshold), start an adaptive re-sampling process (such as incremental verification + hash comparison).

[0108] The technical effect comparison results between the method of the present invention and the traditional method are shown in Table 1.

[0109] Table 1 Technical Effect Comparison Results

[0110] Example 2 For the personalized configuration class of business personnel (S01 special fund appropriation management scenario), the process of implementing collection and conversion using the intelligent collection and conversion method provided in Example 1 is as follows: 1. Rule Configuration: (1)Business personnel drag and drop components such as "project progress verification" and "fund release conditions".

[0111] (2)Set logic: When [Project Progress] is greater than or equal to 80% and [Audit Report] = Pass, connect the two through an "AND" logic component and bind it to the "Fund Appropriation" action component.

[0112] 2. Data collection: (1)The adapter configures the database connection parameters of the financial system interface and the project management platform, and automatically docks with the financial budget management system and the project management platform.

[0113] (2)Internal network data penetrates through the firewall to obtain relevant files or import raw data files.

[0114] (3)Incrementally obtain real-time data: Only collect data rows whose data modification time is later than the last data synchronization time and the MD5 hash value has changed. Adopt a polling method every 30 minutes to obtain incremental data.

[0115] 3. Transformation execution: The engine automatically generates Spark SQL and submits it to the cluster. The SQL content needs to query datasets such as project codes, project names, and project progress whose data modification time is greater than the last data synchronization time according to the rule engine configuration requirements. At the same time, filter out data with a project progress greater than or equal to 80%, synchronize this batch of data, and automatically execute the relevant action components for fund appropriation.

[0116] 4. Result output: (1)Generate a fund appropriation instruction and push it to the relevant execution departments. After the execution departments review it and click the fund appropriation execution button, the system performs instruction execution and data transfer according to the configuration requirements of the relevant action components for fund appropriation.

[0117] (2)Output a visual monitoring dashboard (including the fund-project association view) Example 3 For the automatic generation type (suspected unreasonable management scenario of project fund expenditure), the process of realizing collection and transformation by using the intelligent collection and transformation method provided in Example 1 is as follows: 1. Automatic generation rules: (1)According to the various management requirements of fund expenditure, after deep learning and analysis, automatically generate rules for suspected unreasonableness of a certain project's fund expenditure.

[0118] (2)Generation logic: When [Business Expenses] include fund uses such as publication and literature, and [Violation of Similar Project Recognition], trigger subsequent processing of fund supervision.

[0119] 2. Data collection: (1)Automatically dock with the financial budget management system and the project management platform (2) The intranet data penetrates through the firewall to obtain relevant files or import original data files.

[0120] (3) Incrementally obtain real-time data: Only collect data rows whose modification time is later than the last data synchronization time and the MD5 hash value has changed. Adopt a polling method every 30 minutes to obtain incremental data.

[0121] 3. Conversion and execution: The engine automatically generates Java or Python code blocks. The code content needs to call the fund supervision trigger according to the configuration requirements of the intelligent conversion engine. First, check whether the fund usage contains sensitive keywords ('publication', 'literature', etc.). If there is no violation, the supervision rule is not triggered. If there is a violation, judge the severity according to the violation situation, generate a violation work order, and distribute it to the relevant departments for rectification operations, etc.

[0122] 4. Result output: (1) Trigger the violation condition and push the violation content to the relevant personnel through messages.

[0123] (2) Output a visual monitoring dashboard (including the fund-project association view), violation situations, and relevant management specifications.

[0124] (3) Track the subsequent rectification situation and execute other management requirements.

[0125] By adopting the above technical solutions disclosed in the present invention, the following beneficial effects are obtained: A data intelligent acquisition and conversion method based on a visual rule model provided by the present invention, with a two-way visualization → code conversion mechanism: The "what you see is what you get" rule designer is pioneered, supporting business personnel or technical support personnel to configure complex logics without coding, while retaining the code-level flexibility of developers. A semantic enhancement model in the financial field: An intelligent mapping algorithm based on domain pre-trained BERT to solve the similar semantic gap problem in traditional solutions. A dynamic policy engine and elastic computing: Combining the version management of rules (gray release / rollback) with the self-adaptation of the distributed engine to achieve a minute-level closed-loop of "policy change → rule adjustment → execution and effectiveness".

[0126] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A data intelligent acquisition and conversion method based on a visualization rule model, characterized in that: including the following steps S1. The Visualization Rule Modeling Layer converts the configured rule chain into a JSON-formatted data body, parses the data body into an AST abstract syntax tree, performs field mapping on the AST abstract syntax tree, and then uses the intelligent conversion engine to convert the mapped fields into executable code to generate an execution plan; S2. The Intelligent Collection Layer pulls data from heterogeneous data sources based on the full-volume collection and / or incremental collection methods, cleans the data, and then assigns it to the intelligent conversion engine for parallel processing; S3. The intelligent conversion engine monitors the data lineage graph in real time, monitors the field-level conversion path, and sends signals to the Visualization Rule Modeling Layer or the Intelligent Collection Layer according to the type of exception situation to reconfigure the rule chain or re-collect data.

2. The data intelligent acquisition and conversion method based on the visualization rule model according to claim 1, characterized in that: Step S1 specifically includes the following content S11. The user configures a rule chain in the drag-and-drop rule component library of the Visualization Rule Modeling Layer; S12. The dynamic semantic parsing engine of the Visualization Rule Modeling Layer converts the rule chain into an intermediate representation in JSON format, including component types, parameters, and connection relationships; S13. The dynamic semantic parsing engine of the Visualization Rule Modeling Layer further parses the intermediate representation into an AST abstract syntax tree and optimizes and merges redundant nodes; S14. The intelligent field mapping module of the Visualization Rule Modeling Layer performs field mapping on the AST abstract syntax tree and verifies the mapping error rate; S15. The mapped fields are converted into executable code through the dynamic rule compilation module of the intelligent conversion engine, and Spark SQL or Java classes are generated in real time according to the scenario; S16. The executable code is submitted to the distributed execution framework of the intelligent conversion engine to generate a physical execution plan.

3. The data intelligent acquisition and conversion method based on a visualization rule model according to claim 2, wherein: In step S14, the field mapping process of the intelligent field mapping module is as follows S141. Data input and preprocessing: Receive the source data field and the target structure field, and perform standardization processing on them; S142. Multi-dimensional feature extraction: Use the BERT model to analyze the context similarity described by the source data field and the target structure field, perform structural feature matching on the source data field and the target structure field, and calculate the field name edit distance between the source data field and the target structure field; S143. Similarity calculation and decision-making: Calculate the weighted comprehensive score between the source data field and each target structure field according to the semantic similarity, structural matching degree, and naming similarity between the source data field and the target structure field. When the weighted comprehensive score is greater than or equal to the total score threshold, the source data field is automatically mapped to the corresponding target structure field, and the mapping result is verified. If the verification passes, a mapping scheme is generated and a mapping relationship table is output; If the weighted comprehensive score is less than the total score threshold or the mapping result verification fails, an artificial intervention interface is triggered to display the top three candidate target structure fields; S144. Feedback closed-loop optimization: The user manually corrects the mapping relationship, stores the correction result in the knowledge base, updates the training set of the BERT model, and improves the subsequent matching accuracy.

4. The data intelligent acquisition and conversion method based on the visualization rule model according to claim 1, characterized in that: Step S2 specifically includes the following content S21. The multi-source adapter of the Intelligent Collection Layer pulls data from heterogeneous data sources; S22. For incremental collection, the incremental collection strategy module in the intelligent collection layer performs double verification of timestamps and content hashes on the pulled data; S23. The data cleaning module in the intelligent collection layer performs standardization processing on the pulled data; S24. The standardized data is partitioned according to the primary key hash and distributed to the cluster nodes of the distributed execution framework for parallel processing.

5. The data intelligent acquisition and conversion method based on the visualization rule model according to claim 4, characterized in that: The intelligent collection layer selects the Flink engine or the Spark engine to pull data from heterogeneous data sources according to the data volume of the link. When the data volume is greater than 0.8TB and the latency is less than 5 minutes, the Flink engine is enabled, and the checkpoint interval is set to 300 seconds; when batch processing of data is selected, the Spark engine is enabled, the number of partitions is fixed at 200, and the database automatically distributes the data to 200 physical sub-tables according to the hash value. For frequently queried fields, local indexes are established to avoid degradation of query performance.

6. The data intelligent acquisition and conversion method based on a visualization rule model according to claim 1, characterized in that: Step S3 specifically includes the following content S31. The conversion process visualization module of the intelligent conversion engine monitors the data lineage graph in real time and records the field-level conversion path; S32. If the field mapping error rate is greater than the preset threshold, an optimization signal is sent to the visualization rule modeling layer to trigger reconfiguration of the rule chain; S33. If the data quality is abnormal, an alarm is sent to the intelligent collection layer to start the data re-collection process.

7. The data intelligent acquisition and conversion method based on the visualization rule model according to claim 6, characterized in that: The monitoring process and results of the data lineage graph are displayed using the data lineage tracking interface. The data lineage tracking interface includes a data input layer for displaying the source of the original data, a data processing layer for displaying the conversion logic, an application output layer for displaying the final output, and an external system interaction layer for displaying external interfaces; The data lineage tracking interface can achieve dynamic rendering of the link. According to the data flow path, arrows are automatically drawn to connect; different colors and line styles of connection lines are used to represent normal data links and abnormal data links respectively; at the same time, node indicators are set in the interface, and different colors of indicators are used to represent normal running nodes and nodes with abnormalities.

8. The data intelligent acquisition and conversion method based on the visualization rule model according to claim 7, wherein: The interaction operation process of the data lineage tracking interface is as follows A1. Node metadata viewing: By clicking on a node, the conversion rules, association rules, execution logs, and abnormal data links can be viewed; for abnormal data links, by clicking on the connection line representing the link, the reason for the data link abnormality can be displayed; A2. Time range filtering: By dragging the timeline or entering the start and end times, the interface dynamically refreshes the data lineage graph and only displays the data flow paths for the corresponding time period; by selecting two different time periods, the different links can be highlighted; A3. Abnormality handling and feedback: By clicking on an abnormal node, it can be marked as processed and the corresponding link status is updated; if manual intervention is required, the abnormal details are automatically filled to generate an abnormal work order and pushed to the audit department; A4. Export of data lineage report: By clicking the export button and selecting the export format, a data lineage report can be generated and exported.

9. The data intelligent acquisition and conversion method based on a visualization rule model according to claim 8, wherein: The exported data lineage report includes the complete data lineage graph, key node metadata, and abnormal data link statistics table.

10. The data intelligent acquisition and conversion method based on the visualization rule model according to claim 7, characterized in that: The processing process of abnormal data links in the data lineage tracking interface is as follows B1. Anomaly Detection and Alert: Continuously scan the data link in real time. If an anomaly is detected, a floating alert box will pop up on the data lineage tracking interface, indicating the number of detected abnormal data links; automatically generate work orders for abnormal data links and send them to relevant business departments and technical departments; B2. Manual Intervention and Correction: Click on the field node in the abnormal data link, manually select the target field, and click "Confirm Mapping" to correct the mapping relationship; The correction result is stored in the knowledge base for optimizing subsequent intelligent matching models; B3. Rollback of Historical Versions: Click the "Historical Version" button, select the rule version at a certain point in time, and automatically restore the data lineage graph of that version; the data lineage tracking interface uses color highlighting to show the differences between versions.

Images