Automatic data quality inspection system and method based on large model and data flow arrangement

Abstract

The invention relates to the technical field of big data management and artificial intelligence application, in particular to an automatic data quality inspection system and method based on a big model and data flow orchestration, and the method comprises the following steps: system architecture construction: employing a three-layer architecture design which comprises a data access layer, an intelligent quality inspection layer and a flow orchestration layer; the method has the beneficial effects that the full-link automatic quality inspection of structured and unstructured data is realized by integrating the intelligent analysis capability of a large language model (LLM) and the data processing pipeline of a low-code process arrangement tool. The system combines a rule engine and large model dynamic reasoning, supports customized quality inspection rule generation, abnormal data intelligent identification and quality inspection process adaptive optimization, improves the automation level and complex scene adaptability of data quality detection, and is suitable for data governance scenes of multiple industries such as government, finance, medical treatment, e-commerce and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data governance and artificial intelligence application, in particular to an automatic data quality inspection system and method based on large models and data process arrangement. BACKGROUND

[0002] In the big data era, the data scale grows exponentially, and the data quality directly affects the accuracy of data analysis and decision support. The traditional data quality inspection scheme has the following pain points: 1. Serious dependence on manual work: Quality inspection rules (such as SQL verification and regular expressions) are manually written, and the rule maintenance cost is high when facing complex business scenarios, and it is difficult to cover semantic quality problems (such as field meaning conflict and logical contradiction).

[0003] 2. Insufficient rule flexibility: Static rules are difficult to adapt to changes in business logic. For example, when a new field is added to e-commerce order data, the quality inspection rule needs to be manually updated, which is time-consuming.

[0004] 3. Weak cross-domain quality inspection capability: There is a lack of semantic understanding capability for unstructured data (such as text comments and log files), and traditional rules cannot identify sensitive information leakage and semantic ambiguity in text.

[0005] 4. Low degree of process automation: The data quality inspection process needs to be manually arranged, and the quality inspection link of different data sources needs to be repeatedly developed, making it difficult to achieve dynamic scheduling and resource optimization.

[0006] In the prior art, some schemes attempt to introduce machine learning for anomaly detection, but there are problems such as poor model generalization ability and the need for a large amount of labeled data. The process arrangement tool (Apache NIFI) supports data processing process automation, but lacks intelligent semantic understanding capability. SUMMARY

[0007] The purpose of the present application is to provide an automatic data quality inspection system and method based on large models and data process arrangement to solve the problems raised in the background.

[0008] To achieve the above purpose, the present application provides the following technical scheme: an automatic data quality inspection system based on large models and data process arrangement, comprising: Data access layer: supporting multi-source heterogeneous data collection, covering relational databases, unstructured data, streaming data, and big data, and realizing flexible expansion of data sources through self-defined adapter components; Intelligent quality inspection layer: composed of a large model inference engine, a rule engine, and a quality inspection algorithm library, the large model is responsible for rule generation and semantic understanding, the rule engine performs structured data verification, and the quality inspection algorithm library provides basic algorithms for anomaly detection and consistency verification; Process orchestration layer: based on low-code platform to realize quality inspection process visualization and arrangement, support conditional branching, parallel processing, task scheduling function, and can dynamically adjust the process node according to the quality inspection result.

[0009] Preferably, the data access and preprocessing process of the data access layer includes: Multi-source data collection: use custom extended DataSourceAdapter component to configure data source type, connection parameters, collection frequency; for database collection, provide QueryDatabaseByCursor component to support full, incremental, full+incremental mode, and can configure data source, table, incremental related parameters; Data preprocessing: use DataFormatter component for format standardization, including null value filling, format conversion, special character filtering; use TextSplitter component to split unstructured text data by paragraph, sentence or keyword, and split structured data by row, column or specified rule grouping; automatically extract data field name, type, length, enumeration value meta information to generate data dictionary.

[0010] Preferably, the large model driven intelligent quality inspection process of the intelligent quality inspection layer includes: Quality inspection rule dynamic generation: built-in common quality inspection rule templates and support user-defined template extension; guide large model to generate quality inspection rules for specific business scenarios through Dify platform or custom prompt word arrangement, test rule effectiveness through small-scale data after rule generation, automatically evaluate rule coverage and false positive rate, support user manual adjustment of prompt words to optimize rule generation effect; Unstructured data semantic quality inspection: use large model to analyze text data semantics, identify sensitive information leakage, semantic ambiguity, sentiment tendency abnormality; for system logs, understand log semantics through large model, detect abnormal operation records, compliance problems; Intelligent checking of structured data: based on large model to understand the relationship between tables, automatically generate cross-table association rules; for abnormal detection of numerical fields, large model dynamically calculates reasonable threshold value based on historical data distribution.

[0011] Preferably, the low-code process orchestration and execution process of the process orchestration layer includes: Visual process design: provide drag-and-drop process arrangement interface, support adding data collection, preprocessing, quality inspection rule execution, result output nodes, configure node dependency and data flow; support conditional branching, parallel processing, loop execution control flow; Dynamic process optimization: automatically adjust the process based on quality inspection results, increase pre-check nodes or improve rule execution priority; automatically adjust the number of concurrent threads and allocate computing resources according to the quality inspection task load, support integration with Kubernetes cluster to realize elastic scaling; Process execution and monitoring: task scheduling engine supports three modes of timing scheduling, event triggering and manual triggering; real-time monitoring dashboard displays process execution status, node time consumption, quality inspection pass rate and abnormal data distribution indicators, and supports setting alarm thresholds.

[0012] Preferably, it also includes quality inspection result analysis and feedback optimization process, specifically: Multi-dimensional result visualization: generate quality inspection report, including data quality score, rule execution result statistics, abnormal data details and trend analysis; support interactive filtering of abnormal data; Feedback-driven model iteration: provide an abnormal data labeling interface, users can mark large model mis-detection and missed detection, and the labeled data is automatically used for large model fine-tuning; automatically adjust the large model prompt words according to the feedback results; Abnormal data processing link: for abnormal data found by quality inspection, support automatic correction, marking for processing and triggering downstream process, processing rules can be customized through process arrangement.

[0013] A method of an automatic data quality inspection system based on large model and data process arrangement, comprising the following steps: System architecture building: adopt "three-layer architecture" design, including data access layer, intelligent quality inspection layer and process arrangement layer; Among them, the data access layer supports multi-source heterogeneous data collection, and realizes flexible expansion of data sources through custom adapter components; the intelligent quality inspection layer is composed of large model inference engine, rule engine and quality inspection algorithm library; the process arrangement layer realizes quality inspection process visual arrangement based on low-code platform, supports conditional branching, parallel processing and task scheduling functions, and can dynamically adjust process nodes according to quality inspection results.

[0014] Preferably, the data access and preprocessing steps of the data access layer include: Multi-source data collection: use custom extended DataSourceAdapter component to configure data source type, connection parameters and collection frequency; for database collection, use QueryDatabaseByCursor component, support full, incremental and full + incremental modes, configure data source, table and incremental related parameters, and set incremental alarm threshold; Data preprocessing: format standardization: use the DataFormatter component to clean the data, including null value filling, format conversion, special character filtering; data splitting: for unstructured text data, use the TextSplitter component to split by paragraph, sentence or keyword; for structured data, support splitting by row, column or grouping according to specified rules; Metadata extraction: automatically extract data field names, types, lengths, and enumeration value metadata to generate a data dictionary.

[0015] Preferably, the large model driven intelligent inspection step of the intelligent inspection layer includes: Dynamic generation of inspection rules: use the rule template library, built-in common inspection rule templates and support user-defined template extension; through Dify platform or custom prompt word arrangement, guide the large model to generate inspection rules for specific business scenarios; after the large model generates the rules, test the rules for effectiveness through small-scale data, automatically evaluate the rule coverage and false positive rate, and support user manual adjustment of prompt words to optimize rule generation effect; Unstructured data semantic inspection: use the large model to analyze the semantics of text data, identify sensitive information leakage, semantic ambiguity, and emotional tendency abnormalities; for system logs, understand the semantics of the logs through the large model, and detect abnormal operation records and compliance issues; Intelligent checking of structured data: based on the large model understanding of inter-table relationships, automatically generate cross-table association rules; for abnormal detection of numerical fields, the large model dynamically calculates reasonable thresholds based on historical data distribution.

[0016] Preferably, the low-code process orchestration and execution steps of the process orchestration layer include: Visual process design: provides a drag-and-drop process orchestration interface, supports adding data collection, preprocessing, inspection rule execution, and result output nodes, and configures the dependency relationship and data flow between nodes; supports conditional branching, parallel processing, and loop execution control flow; Dynamic process optimization: automatically adjust the process based on the inspection results, add pre-check nodes or improve the priority of rule execution; automatically adjust the number of concurrent threads and allocate computing resources according to the inspection task load, and support integration with Kubernetes cluster to realize elastic scaling; Process execution and monitoring: the task scheduling engine supports three modes of timing scheduling, event triggering, and manual triggering; real-time monitoring dashboard displays process execution status, node time consumption, inspection pass rate, and abnormal data distribution indicators, and supports setting alarm thresholds.

[0017] Preferably, it also includes the steps of inspection result analysis and feedback optimization, specifically: Multi-dimensional result visualization: generate quality inspection reports, including data quality scores, rule execution result statistics, abnormal data details, and trend analysis; support interactive filtering of abnormal data; Feedback-driven model iteration: provide an abnormal data labeling interface, users mark large model mis-detection and missed detection, and the labeled data is automatically used for large model fine-tuning; automatically adjust the large model prompt word according to the feedback result; Abnormal data processing link: for abnormal data found by quality inspection, support automatic correction, marking for processing, and triggering downstream processes, and the processing rules can be customized through process arrangement.

[0018] Compared with the prior art, the beneficial effects of the present application are: The automatic data quality inspection system and method based on a large model and data process arrangement provided by the present application realizes full-link automatic quality inspection of structured and unstructured data by integrating the intelligent analysis capability of a large language model (LLM) and the data processing pipeline of a low-code process arrangement tool. The system combines a rule engine and dynamic reasoning of a large model to support custom quality inspection rule generation, intelligent identification of abnormal data, and adaptive optimization of quality inspection processes, thereby improving the automation level and adaptability to complex scenarios of data quality detection, and being suitable for data governance scenarios in multiple industries such as government, finance, medical treatment, and e-commerce. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The figure is a system architecture diagram of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution of the present application clear, complete and the advantages more clear and obvious, the embodiments of the present application are further described in detail below in combination with the drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present application, not all embodiments, and are only used to explain the embodiments of the present application, and do not limit the embodiments of the present application. All other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] The present application provides a technical solution: an automatic data quality inspection system and method based on a large model and data process arrangement, the detailed steps are as follows: (I) System architecture design The present system adopts a "three-layer architecture" design, including: (1) Data access layer: Supports multi-source heterogeneous data collection, including relational databases (MySQL, PostgreSQL), unstructured data (log files, JSON documents), streaming data (Kafka, Flink), big data (Hive, HDFS), etc. Flexible expansion of data sources is achieved through custom adapter components.

[0022] (2) Intelligent quality inspection layer: Core layer composed of large model inference engine, rule engine, and quality inspection algorithm library. The large model is responsible for rule generation and semantic understanding; the rule engine performs structured data verification; and the quality inspection algorithm library provides basic algorithms such as anomaly detection and consistency verification.

[0023] (3) Process orchestration layer: Based on a low-code platform, the quality inspection process is visually arranged, supporting conditional branching, parallel processing, task scheduling, and other functions. The process nodes can be dynamically adjusted according to the quality inspection results.

[0024] (II) Data access and preprocessing (1) Multi-source data collection 1. Use the custom extended DataSourceAdapter component to support configuration of data source types (database, file system, API interface), connection parameters, collection frequency, etc.

[0025] 2. For database collection, provide the QueryDatabaseByCursor component, which supports full, incremental, and full + incremental modes. Configuration examples are as follows: Data source configuration: Specify database type, IP address, port, username, and password; table configuration: select target table, field mapping relationship, and filter conditions (such as WHERE create_time>'2025-01-01'); incremental configuration: implement incremental data capture based on timestamp or primary key, and set incremental alarm threshold (such as triggering an alarm when the single incremental data volume exceeds 100,000).

[0026] (2) Data preprocessing 1. Format standardization: Use the DataFormatter component to clean the data, including null value filling (mean, mode, custom value), format conversion (date format unification, numerical type standardization), and special character filtering.

[0027] 2. Data splitting: For unstructured text data, use the TextSplitter component to split by paragraph, sentence, or keyword for easy large model block processing; for structured data, support row, column splitting, or grouping according to specified rules (such as date partitioning).

[0028] 3. Metadata extraction: Automatically extract metadata such as data field names, types, lengths, enumeration values, etc., generate data dictionaries, and provide a basis for subsequent quality inspection rule generation.

[0029] (Three) Intelligent quality inspection driven by large models (1) Dynamic generation of quality inspection rules 1. Rule template library: Built-in common quality inspection rule templates (such as uniqueness verification, non-empty verification, value range verification, logical consistency verification), support user-defined template extension.

[0030] 2. Large model prompt engineering: Through Dify platform or custom prompt word arrangement, guide the large model to generate quality inspection rules for specific business scenarios. Example prompt words: "Please generate quality inspection rules according to the following data table structure: table name: order table, field: order number (unique identifier), order time (date type), amount (numeric type, >0), status (enumeration value: to be paid, paid, canceled), requirements: generate 5 quality inspection rules, including uniqueness, non-empty, value range, enumeration value verification and one business logic verification (such as 'paid order amount must be >0'), rules returned in JSON format, including rule ID, description, execution priority." 3. Rule verification and optimization: After the large model generates rules, test the rules with small-scale data to verify their effectiveness, automatically evaluate rule coverage and false positive rate, and support user manual adjustment of prompt words to optimize rule generation effect.

[0031] (2) Unstructured data semantic quality inspection 1. Text classification and anomaly detection: Use large models to analyze the semantics of text data (such as customer service chat records, user comments), identify sensitive information leaks (such as ID number, bank card number), semantic ambiguity (such as contradictory expressions such as "order canceled" and "not canceled"), emotional tendency abnormalities, etc.

[0032] 2. Log file compliance check: For system logs, understand the semantics of the logs through large models, detect abnormal operation records (such as unauthorized access, high-frequency error requests), compliance issues (such as not recording key operation logs).

[0033] (3) Structured data intelligent verification 1. Cross-table association verification: Based on the large model's understanding of the relationship between tables, automatically generate cross-table association rules (such as "the user ID in the order table must exist in the user table"), replacing the traditional manual JOIN statement writing method.

[0034] 2. Dynamic threshold calculation: For anomaly detection on numerical fields (e.g., transaction amount fluctuation), the large model can dynamically calculate a reasonable threshold (e.g., 3 times the standard deviation range) based on historical data distribution, avoiding the problem that static thresholds cannot adapt to business fluctuations.

[0035] (Four) Low-code process orchestration and execution (1) Visual process design Provide a drag-and-drop process orchestration interface, support adding data collection, preprocessing, quality inspection rule execution, result output nodes, and configure the dependency relationship and data flow between nodes.

[0036] Support conditional branching (e.g., "if quality inspection fails, enter the manual review process"), parallel processing (e.g., perform multi-dimensional quality inspection simultaneously), and loop execution (e.g., repeatedly quality inspect batch data).

[0037] (2) Dynamic process optimization Process adjustment based on quality inspection results: If a certain type of data frequently triggers the same quality inspection rule failure, the system automatically adjusts the process, adds a pre-check node, or increases the execution priority of the rule.

[0038] Resource dynamic scheduling: According to the quality inspection task load, automatically adjust the number of concurrent threads, allocate computing resources (such as CPU, memory), and support integration with Kubernetes cluster to achieve elastic scaling.

[0039] (3) Process execution and monitoring Task scheduling engine: Support three modes of timing scheduling (e.g., perform full-quality inspection every morning), event triggering (e.g., automatically trigger incremental quality inspection after data update), and manual triggering.

[0040] Real-time monitoring dashboard: Display process execution status, node time consumption, quality inspection pass rate, abnormal data distribution, and other indicators, and support setting alarm thresholds (e.g., send SMS or email notifications when quality inspection time consumption exceeds 30 minutes).

[0041] (Five) Quality inspection result analysis and feedback optimization (1) Multi-dimensional result visualization Generate quality inspection reports, including data quality scores, rule execution result statistics, abnormal data details (such as fields, error types, and occurrence frequencies), and trend analysis. Support interactive filtering (e.g., filter abnormal data by business line, data type, and error level), making it easy to locate the root cause of the problem.

[0042] (2) Feedback-driven model iteration Human annotation and feedback: Provide an abnormal data annotation interface, users can mark large model false positives and false negatives, and annotated data is automatically used for large model fine-tuning to improve subsequent quality inspection accuracy. Prompt optimization: Based on feedback results, automatically adjust large model prompts, for example, if a rule generation misses certain business logic, the system automatically adds related constraints to the prompt.

[0043] (3) Abnormal data processing link For abnormal data found by quality inspection, three processing methods are supported: automatic correction (such as filling missing values, correcting format errors), marking for processing, and triggering downstream processes (such as pushing abnormal orders to manual review). Processing rules can be customized through process orchestration.

[0044] (Six) Key technology component implementation (1) Large model integration component (LLMConnector) Encapsulate mainstream large model APIs (such as OpenAI GPT-4, Anthropic Claude), support model switching, temperature parameter adjustment, maximum token limit configuration, etc. Implement prompt template management, historical dialogue record caching to reduce repeated request overhead and improve large model call efficiency.

[0045] (2) Process orchestration engine (FlowOrchestrator) Based on state machine to realize process node scheduling and state management, support process version control (such as rollback to historical version), breakpoint resume (continue execution from the last successful node after interruption). Provide RESTful API interface, support external system call to trigger quality inspection process, query process state.

[0046] (3) Rule engine (RuleEngine) Supports SQL expressions, regular expressions, custom scripts (Python / Java), and other rule execution methods, compatible with traditional quality inspection rules and large model generated rules.

[0047] Implement rule priority management, conflict detection (such as when two rules have conflicting validation conditions for the same field, automatically alert).

[0048] (Seven) Typical application scenario process example Take e-commerce order data quality inspection as an example, the complete process is as follows: 1. Data collection: Incrementally collect order table data every morning through the QueryDatabaseByCursor component, and store it by date partition.

[0049] 2. Preprocessing: clean null values with DataFormatter, split order note text into short sentences with TextSplitter.

[0050] 3. Rule generation: large model generates basic rules (e.g., order number is unique, amount > 0) according to order table structure, and generates business rules (e.g., "cancelled orders cannot have payment records") combined with e-commerce business knowledge.

[0051] 4. Parallel quality inspection: Structured field verification: non-empty, value range, uniqueness verification is performed through the rule engine; text semantic analysis: large model analyzes sensitive information (e.g., user mobile number) and complaint keywords in the note text; cross-table association: check if the order user ID exists in the user table and if the order item ID exists in the item table.

[0052] 5. Result summary: merge quality inspection results of each dimension, generate order quality score, and mark high-risk orders (e.g., amount anomaly + note contains complaint word).

[0053] 6. Abnormal processing: high-risk orders are automatically pushed to customer service for review, and low-risk abnormal data is automatically corrected (e.g., fill in missing order status).

[0054] 7. Feedback optimization: customer service review results are fed back to the system to fine-tune the large model's business rule generation capabilities.

[0055] Although embodiments of the present application have been shown and described, it is to be understood that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An automated data quality inspection system based on large models and data flow orchestration, characterized by: include: Data access layer: Supports the acquisition of multi-source heterogeneous data, covering relational databases, unstructured data, streaming data, and big data, and enables flexible expansion of data sources through custom adapter components; Intelligent quality inspection layer: It consists of a large model inference engine, a rule engine, and a quality inspection algorithm library. The large model is responsible for rule generation and semantic understanding, the rule engine performs structured data verification, and the quality inspection algorithm library provides basic algorithms for anomaly detection and consistency verification. Workflow orchestration layer: Based on a low-code platform, it realizes visual orchestration of quality inspection processes, supports conditional branching, parallel processing, and task scheduling functions, and can dynamically adjust process nodes according to quality inspection results.

2. The automated data quality inspection system based on large model and data flow orchestration according to claim 1, characterized in that: The data access and preprocessing process at the data access layer includes: Multi-source data acquisition: Use the custom extended DataSourceAdapter component to configure the data source type, connection parameters, and acquisition frequency; for database acquisition, the QueryDatabaseByCursor component is provided to support full, incremental, and full + incremental modes, and the data source, table, and incremental related parameters can be configured; Data preprocessing: The DataFormatter component is used for format standardization, including null value filling, format conversion, and special character filtering; the TextSplitter component is used to split unstructured text data by paragraph, sentence, or keyword, and to split structured data by row, column, or group it according to specified rules; data field names, types, lengths, and enumeration values ​​are automatically extracted to generate a data dictionary.

3. The automated data quality inspection system based on large model and data flow orchestration according to claim 2, characterized in that: The intelligent quality inspection process driven by a large model in the intelligent quality inspection layer includes: Dynamic generation of quality inspection rules: Built-in common quality inspection rule templates and support for user-defined template extensions; guides large models to generate quality inspection rules for specific business scenarios through the Dify platform or custom prompt words; after generating rules, the effectiveness of the rules is tested through small-scale data, and the rule coverage and false alarm rate are automatically evaluated; users can manually adjust prompt words to optimize the rule generation effect. Semantic quality inspection of unstructured data: Using large models to perform semantic analysis on text data to identify issues such as sensitive information leakage, semantic ambiguity, and abnormal sentiment; for system logs, using large models to understand log semantics and detect abnormal operation records and compliance issues; Intelligent validation of structured data: Based on the understanding of inter-table relationships using a large model, cross-table association rules are automatically generated; for anomaly detection of numerical fields, the large model dynamically calculates reasonable thresholds based on historical data distribution.

4. The automated data quality inspection system based on large model and data flow orchestration according to claim 3, characterized in that: The low-code process orchestration and execution process of the process orchestration layer includes: Visual workflow design: Provides a drag-and-drop workflow orchestration interface, supports adding data acquisition, preprocessing, quality inspection rule execution, and result output nodes, and configuring dependencies and data flow between nodes; supports conditional branching, parallel processing, and loop execution control flow; Dynamic process optimization: Automatically adjusts the process based on quality inspection results, adding pre-verification nodes or increasing the execution priority of rules; automatically adjusts the number of concurrent threads and allocates computing resources according to the load of quality inspection tasks, and supports integration with Kubernetes clusters to achieve elastic scaling. Process execution and monitoring: The task scheduling engine supports three modes: scheduled, event-triggered, and manual-triggered. The real-time monitoring dashboard displays the process execution status, node time, quality inspection pass rate, and abnormal data distribution indicators, and supports setting alarm thresholds.

5. The automated data quality inspection system based on large model and data flow orchestration according to claim 4, characterized in that: It also includes the process of quality inspection result analysis and feedback optimization, specifically: Multi-dimensional result visualization: Generates quality inspection reports, including data quality scores, statistics of each rule's execution results, details of abnormal data, and trend analysis; supports interactive filtering of abnormal data. Feedback-driven model iteration: Provides an interface for anomaly data annotation, allowing users to mark false positives and false negatives in the large model. The annotated data is automatically used for fine-tuning the large model; the prompt words of the large model are automatically adjusted based on feedback results. Abnormal data processing chain: For abnormal data discovered by quality inspection, three processing methods are supported: automatic error correction, marking for processing, and triggering downstream processes. Processing rules can be customized through process orchestration.

6. An automated data quality inspection method based on large models and data flow orchestration, characterized in that: The system described in claim 5 is used in the following steps: System architecture: A three-tier architecture is adopted, including a data access layer, an intelligent quality inspection layer, and a process orchestration layer; The data access layer supports the acquisition of multi-source heterogeneous data and enables flexible expansion of data sources through custom adapter components; the intelligent quality inspection layer consists of a large model inference engine, a rule engine, and a quality inspection algorithm library; the process orchestration layer realizes the visual orchestration of the quality inspection process based on a low-code platform, supports conditional branching, parallel processing, and task scheduling functions, and can dynamically adjust process nodes according to the quality inspection results.

7. A method according to claim 6, characterized in that: The data access and preprocessing steps at the data access layer include: Multi-source data acquisition: Use the custom extended DataSourceAdapter component to configure the data source type, connection parameters, and acquisition frequency; for database acquisition, use the QueryDatabaseByCursor component, which supports full, incremental, and full + incremental modes, configure the data source, table, incremental related parameters, and set the incremental alarm threshold; Data preprocessing: Format standardization: Data is cleaned using the DataFormatter component, including null padding, format conversion, and special character filtering; Data splitting: For unstructured text data, the TextSplitter component is used to split the data by paragraph, sentence, or keyword; For structured data, it supports splitting by row, column, or grouping by specified rules. Metadata extraction: Automatically extracts data field names, types, lengths, and enumeration value metadata to generate a data dictionary.

8. A method according to claim 6, characterized in that: The intelligent quality inspection steps driven by the large model of the intelligent quality inspection layer include: Dynamic generation of quality inspection rules: Utilizing a rule template library, common quality inspection rule templates are built-in and user-defined template extensions are supported; through the Dify platform or custom prompt words, large models are guided to generate quality inspection rules for specific business scenarios; after the large model generates rules, the effectiveness of the rules is tested with small-scale data, and the rule coverage and false alarm rate are automatically evaluated. Users can also manually adjust prompt words to optimize the rule generation effect. Semantic quality inspection of unstructured data: Using large models to perform semantic analysis on text data to identify issues such as sensitive information leakage, semantic ambiguity, and abnormal sentiment; for system logs, using large models to understand log semantics and detect abnormal operation records and compliance issues; Intelligent validation of structured data: Based on the understanding of inter-table relationships using a large model, cross-table association rules are automatically generated; for anomaly detection of numerical fields, the large model dynamically calculates reasonable thresholds based on historical data distribution.

9. A method according to claim 6, characterized in that: The low-code process orchestration and execution steps of the process orchestration layer include: Visual workflow design: Provides a drag-and-drop workflow orchestration interface, supports adding data acquisition, preprocessing, quality inspection rule execution, and result output nodes, and configuring dependencies and data flow between nodes; supports conditional branching, parallel processing, and loop execution control flow; Dynamic process optimization: Automatically adjusts the process based on quality inspection results, adding pre-verification nodes or increasing the execution priority of rules; automatically adjusts the number of concurrent threads and allocates computing resources according to the load of quality inspection tasks, and supports integration with Kubernetes clusters to achieve elastic scaling. Process execution and monitoring: The task scheduling engine supports three modes: scheduled, event-triggered, and manual-triggered. The real-time monitoring dashboard displays the process execution status, node time, quality inspection pass rate, and abnormal data distribution indicators, and supports setting alarm thresholds.

10. A method according to claim 6, characterized in that: It also includes quality inspection result analysis and feedback optimization steps, specifically: Multi-dimensional result visualization: Generates quality inspection reports, including data quality scores, statistics of each rule's execution results, details of abnormal data, and trend analysis; supports interactive filtering of abnormal data. Feedback-driven model iteration: Provides an interface for anomaly data annotation, allowing users to mark false positives and false negatives in the large model. The annotated data is automatically used for fine-tuning the large model; the prompt words of the large model are automatically adjusted based on feedback results. Abnormal data processing chain: For abnormal data discovered by quality inspection, three processing methods are supported: automatic error correction, marking for processing, and triggering downstream processes. Processing rules can be customized through process orchestration.

Images