Automated Data Quality Inspection System and Method Based on Large Model and Data Flow Orchestration

Abstract

This invention relates to the fields of big data governance and artificial intelligence application technology, specifically to an automated data quality inspection system and method based on large models and data flow orchestration, comprising the following steps: System architecture construction: A three-layer architecture design is adopted, including a data access layer, an intelligent quality inspection layer, and a flow orchestration layer; The beneficial effects are: By integrating the intelligent analysis capabilities of large language models (LLM) with the data processing pipeline of low-code flow orchestration tools, full-link automated quality inspection of structured and unstructured data is achieved. The system combines a rule engine with dynamic reasoning of large models, supports the generation of custom quality inspection rules, intelligent identification of abnormal data, and adaptive optimization of the quality inspection process, improving the automation level and adaptability to complex scenarios of data quality inspection, and is suitable for data governance scenarios in multiple industries such as government, finance, healthcare, and e-commerce.

Description

Technical Field

[0001] This invention relates to the fields of big data governance and artificial intelligence application technology, specifically to an automated data quality inspection system and method based on large models and data flow orchestration. Background Technology

[0002] In the era of big data, the scale of data is growing exponentially, and data quality directly affects the accuracy of data analysis and decision support. Traditional data quality inspection solutions have the following pain points:

[0003] 1. Heavy reliance on manual intervention: It relies on manually writing quality inspection rules (such as SQL validation and regular expressions). When facing complex business scenarios, the rule maintenance cost is high, and it is difficult to cover semantic quality issues (such as field meaning conflicts and logical contradictions).

[0004] 2. Insufficient rule flexibility: Static rules are difficult to adapt to changes in business logic. For example, when a new field is added to e-commerce order data, the quality inspection rules need to be updated manually, which is not timely.

[0005] 3. Weak cross-domain quality inspection capabilities: It lacks the ability to understand the semantics of unstructured data (such as text comments and log files), and traditional rules cannot identify problems such as the leakage of sensitive information and semantic ambiguity in the text.

[0006] 4. Low degree of process automation: Data quality inspection processes need to be manually arranged, and quality inspection links for different data sources are repeatedly developed, making it difficult to achieve dynamic scheduling and resource optimization.

[0007] In existing technologies, some solutions attempt to introduce machine learning for anomaly detection, but they suffer from problems such as poor model generalization ability and the need for a large amount of labeled data; while workflow orchestration tools (Apache NIFI) support the automation of data processing workflows, they lack intelligent semantic understanding capabilities. Summary of the Invention

[0008] The purpose of this invention is to provide an automated data quality inspection system and method based on large models and data flow orchestration to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: an automated data quality inspection system based on large models and data flow orchestration, comprising:

[0010] Data access layer: Supports the acquisition of multi-source heterogeneous data, covering relational databases, unstructured data, streaming data, and big data, and enables flexible expansion of data sources through custom adapter components;

[0011] Intelligent quality inspection layer: It consists of a large model inference engine, a rule engine, and a quality inspection algorithm library. The large model is responsible for rule generation and semantic understanding, the rule engine performs structured data verification, and the quality inspection algorithm library provides basic algorithms for anomaly detection and consistency verification.

[0012] Workflow orchestration layer: Based on a low-code platform, it realizes visual orchestration of quality inspection processes, supports conditional branching, parallel processing, and task scheduling functions, and can dynamically adjust process nodes according to quality inspection results.

[0013] Preferably, the data access and preprocessing process at the data access layer includes:

[0014] Multi-source data acquisition: Use the custom extended DataSourceAdapter component to configure the data source type, connection parameters, and acquisition frequency; for database acquisition, the QueryDatabaseByCursor component is provided to support full, incremental, and full + incremental modes, and the data source, table, and incremental related parameters can be configured;

[0015] Data preprocessing: The DataFormatter component is used for format standardization, including null value filling, format conversion, and special character filtering; the TextSplitter component is used to split unstructured text data by paragraph, sentence, or keyword, and to split structured data by row, column, or group it according to specified rules; data field names, types, lengths, and enumeration values ​​are automatically extracted to generate a data dictionary.

[0016] Preferably, the intelligent quality inspection process driven by a large model in the intelligent quality inspection layer includes:

[0017] Dynamic generation of quality inspection rules: Built-in common quality inspection rule templates and support for user-defined template extensions; guides large models to generate quality inspection rules for specific business scenarios through the Dify platform or custom prompt words; after generating rules, the effectiveness of the rules is tested through small-scale data, and the rule coverage and false alarm rate are automatically evaluated; users can manually adjust prompt words to optimize the rule generation effect.

[0018] Semantic quality inspection of unstructured data: Using large models to perform semantic analysis on text data to identify issues such as sensitive information leakage, semantic ambiguity, and abnormal sentiment; for system logs, using large models to understand log semantics and detect abnormal operation records and compliance issues;

[0019] Intelligent validation of structured data: Based on the understanding of inter-table relationships using a large model, cross-table association rules are automatically generated; for anomaly detection of numerical fields, the large model dynamically calculates reasonable thresholds based on historical data distribution.

[0020] Preferably, the low-code process orchestration and execution process of the process orchestration layer includes:

[0021] Visual workflow design: Provides a drag-and-drop workflow orchestration interface, supports adding data acquisition, preprocessing, quality inspection rule execution, and result output nodes, and configuring dependencies and data flow between nodes; supports conditional branching, parallel processing, and loop execution control flow;

[0022] Dynamic process optimization: Automatically adjusts the process based on quality inspection results, adding pre-verification nodes or increasing the execution priority of rules; automatically adjusts the number of concurrent threads and allocates computing resources according to the load of quality inspection tasks, and supports integration with Kubernetes clusters to achieve elastic scaling.

[0023] Process execution and monitoring: The task scheduling engine supports three modes: scheduled, event-triggered, and manual-triggered. The real-time monitoring dashboard displays the process execution status, node time, quality inspection pass rate, and abnormal data distribution indicators, and supports setting alarm thresholds.

[0024] Preferably, it also includes a quality inspection result analysis and feedback optimization process, specifically:

[0025] Multi-dimensional result visualization: Generates quality inspection reports, including data quality scores, statistics of each rule's execution results, details of abnormal data, and trend analysis; supports interactive filtering of abnormal data.

[0026] Feedback-driven model iteration: Provides an interface for anomaly data annotation, allowing users to mark false positives and false negatives in the large model. The annotated data is automatically used for fine-tuning the large model; the prompt words of the large model are automatically adjusted based on feedback results.

[0027] Abnormal data processing chain: For abnormal data discovered by quality inspection, three processing methods are supported: automatic error correction, marking for processing, and triggering downstream processes. The processing rules can be customized through process orchestration.

[0028] A method for an automated data quality inspection system based on large models and data flow orchestration includes the following steps:

[0029] System architecture: A three-tier architecture is adopted, including a data access layer, an intelligent quality inspection layer, and a process orchestration layer;

[0030] The data access layer supports the acquisition of multi-source heterogeneous data and enables flexible expansion of data sources through custom adapter components; the intelligent quality inspection layer consists of a large model inference engine, a rule engine, and a quality inspection algorithm library; the process orchestration layer realizes the visual orchestration of the quality inspection process based on a low-code platform, supports conditional branching, parallel processing, and task scheduling functions, and can dynamically adjust process nodes according to the quality inspection results.

[0031] Preferably, the data access and preprocessing steps of the data access layer include:

[0032] Multi-source data acquisition: Use the custom extended DataSourceAdapter component to configure the data source type, connection parameters, and acquisition frequency; for database acquisition, use the QueryDatabaseByCursor component, which supports full, incremental, and full + incremental modes, configure the data source, table, incremental related parameters, and set incremental alarm thresholds;

[0033] Data preprocessing: Format standardization: Data is cleaned using the DataFormatter component, including null padding, format conversion, and special character filtering; Data splitting: For unstructured text data, the TextSplitter component is used to split the data by paragraph, sentence, or keyword; For structured data, it supports splitting by row, column, or grouping by specified rules.

[0034] Metadata extraction: Automatically extracts data field names, types, lengths, and enumeration value metadata to generate a data dictionary.

[0035] Preferably, the intelligent quality inspection steps driven by the large model of the intelligent quality inspection layer include:

[0036] Dynamic generation of quality inspection rules: Utilizing a rule template library, common quality inspection rule templates are built-in and user-defined template extensions are supported; through the Dify platform or custom prompt words, large models are guided to generate quality inspection rules for specific business scenarios; after the large model generates rules, the effectiveness of the rules is tested with small-scale data, and the rule coverage and false alarm rate are automatically evaluated. Users can also manually adjust prompt words to optimize the rule generation effect.

[0037] Semantic quality inspection of unstructured data: Using large models to perform semantic analysis on text data to identify issues such as sensitive information leakage, semantic ambiguity, and abnormal sentiment; for system logs, using large models to understand log semantics and detect abnormal operation records and compliance issues;

[0038] Intelligent validation of structured data: Based on the understanding of inter-table relationships using a large model, cross-table association rules are automatically generated; for anomaly detection of numerical fields, the large model dynamically calculates reasonable thresholds based on historical data distribution.

[0039] Preferably, the low-code process orchestration and execution steps of the process orchestration layer include:

[0040] Visual workflow design: Provides a drag-and-drop workflow orchestration interface, supports adding data acquisition, preprocessing, quality inspection rule execution, and result output nodes, and configuring dependencies and data flow between nodes; supports conditional branching, parallel processing, and loop execution control flow;

[0041] Dynamic process optimization: Automatically adjusts the process based on quality inspection results, adding pre-verification nodes or increasing the execution priority of rules; automatically adjusts the number of concurrent threads and allocates computing resources according to the load of quality inspection tasks, and supports integration with Kubernetes clusters to achieve elastic scaling.

[0042] Process execution and monitoring: The task scheduling engine supports three modes: scheduled, event-triggered, and manual-triggered. The real-time monitoring dashboard displays the process execution status, node time, quality inspection pass rate, and abnormal data distribution indicators, and supports setting alarm thresholds.

[0043] Preferably, it also includes quality inspection result analysis and feedback optimization steps, specifically:

[0044] Multi-dimensional result visualization: Generates quality inspection reports, including data quality scores, statistics of each rule's execution results, details of abnormal data, and trend analysis; supports interactive filtering of abnormal data.

[0045] Feedback-driven model iteration: Provides an interface for anomaly data annotation, allowing users to mark false positives and false negatives in the large model. The annotated data is automatically used for fine-tuning the large model; the prompt words of the large model are automatically adjusted based on feedback results.

[0046] Abnormal data processing chain: For abnormal data discovered by quality inspection, three processing methods are supported: automatic error correction, marking for processing, and triggering downstream processes. The processing rules can be customized through process orchestration.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention proposes an automated data quality inspection system and method based on large models and data flow orchestration. By integrating the intelligent analysis capabilities of Large Language Models (LLM) with a low-code flow orchestration tool's data processing pipeline, it achieves end-to-end automated quality inspection of both structured and unstructured data. The system combines a rule engine with dynamic reasoning from large models, supporting the generation of custom quality inspection rules, intelligent identification of abnormal data, and adaptive optimization of the quality inspection process. This enhances the automation level and adaptability to complex scenarios in data quality inspection, making it suitable for data governance scenarios across various industries, including government, finance, healthcare, and e-commerce. Attached Figure Description

[0049] Figure 1 This is a system architecture diagram of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention provides a technical solution: an automated data quality inspection system and method based on large models and data flow orchestration, the detailed steps of which are as follows:

[0052] (I) System Architecture Design

[0053] This system adopts a three-tier architecture design, including:

[0054] (1) Data access layer: Supports the collection of multi-source heterogeneous data, including relational databases (MySQL, PostgreSQL), unstructured data (log files, JSON documents), streaming data (Kafka, Flink), big data (Hive, HDFS), etc., and realizes flexible expansion of data sources through custom adapter components.

[0055] (2) Intelligent Quality Inspection Layer: The core layer, consisting of a large model inference engine, a rule engine, and a quality inspection algorithm library. The large model is responsible for rule generation and semantic understanding; the rule engine performs structured data verification; and the quality inspection algorithm library provides basic algorithms such as anomaly detection and consistency verification.

[0056] (3) Process orchestration layer: Based on the low-code platform, the quality inspection process is visualized and orchestrated, supporting functions such as conditional branching, parallel processing, and task scheduling. The process nodes can be dynamically adjusted according to the quality inspection results.

[0057] (II) Data Access and Preprocessing

[0058] (1) Multi-source data acquisition

[0059] 1. Use the custom extended DataSourceAdapter component to support configuring data source type (database, file system, API interface), connection parameters, collection frequency, etc.

[0060] 2. For database data collection, the QueryDatabaseByCursor component is provided, supporting full, incremental, and full + incremental modes. A configuration example is shown below:

[0061] Data source configuration: Specify the database type, IP address, port, username, and password; Table configuration: Select the target table, field mapping relationship, and filter conditions (e.g., WHERE create_time>'2025-01-01'); Incremental configuration: Implement incremental data capture based on timestamps or primary keys, and set incremental alarm thresholds (e.g., trigger an alarm when the amount of incremental data exceeds 100,000 records in a single instance).

[0062] (2) Data preprocessing

[0063] 1. Format standardization: Use the DataFormatter component to clean the data, including filling in null values ​​(mean, mode, custom values), format conversion (unifying date formats, standardizing numeric types), filtering special characters, etc.

[0064] 2. Data Splitting: For unstructured text data, the TextSplitter component is used to split the data by paragraph, sentence, or keyword, which facilitates the block processing of large models; for structured data, it supports splitting by row, column, or grouping by specified rules (such as partitioning by date).

[0065] 3. Metadata Extraction: Automatically extracts metadata such as data field names, types, lengths, and enumeration values ​​to generate a data dictionary, providing a foundation for subsequent quality inspection rule generation.

[0066] (III) Intelligent quality inspection driven by large models

[0067] (1) Dynamic generation of quality inspection rules

[0068] 1. Rule Template Library: Built-in common quality inspection rule templates (such as uniqueness check, non-empty check, value range check, logical consistency check), and supports user-defined template extensions.

[0069] 2. Large Model Hint Project: Guide large models to generate quality inspection rules for specific business scenarios through the Dify platform or custom hint word arrangement. Example hint word: "Please generate quality inspection rules based on the following data table structure: Table name: Order table, Fields: Order number (unique identifier), Order time (date type), Amount (numeric, >0), Status (enumeration values: pending payment, paid, canceled), Requirements: Generate 5 quality inspection rules, including uniqueness, non-null, value range, enumeration value validation, and one business logic validation (e.g., 'Paid order amount must be >0'). The rules should be returned in JSON format, including rule ID, description, and execution priority."

[0070] 3. Rule Validation and Optimization: After the large model generates rules, the effectiveness of the rules is tested with small-scale data. The rule coverage and false alarm rate are automatically evaluated, and users can manually adjust the prompt words to optimize the rule generation effect.

[0071] (2) Semantic quality inspection of unstructured data

[0072] 1. Text Classification and Anomaly Detection: Utilize large models to perform semantic analysis on text data (such as customer service chat logs and user comments) to identify issues such as sensitive information leakage (e.g., ID card numbers, bank card numbers), semantic ambiguity (e.g., contradictory statements such as "order has been cancelled" and "not cancelled"), and abnormal sentiment.

[0073] 2. Log file compliance check: For system logs, use a large model to understand the log semantics and detect abnormal operation records (such as unauthorized access, high frequency of erroneous requests) and compliance issues (such as failure to record critical operation logs).

[0074] (3) Intelligent verification of structured data

[0075] 1. Cross-table join validation: Based on the understanding of the relationship between tables in the large model, it automatically generates cross-table join rules (such as "the user ID in the order table must exist in the user table"), replacing the traditional method of manually writing JOIN statements.

[0076] 2. Dynamic threshold calculation: For anomaly detection of numerical fields (such as fluctuations in transaction amount), large models can dynamically calculate reasonable thresholds (such as a range of 3 times the standard deviation) based on the distribution of historical data, avoiding the problem that static thresholds cannot adapt to business fluctuations.

[0077] (iv) Low-code process orchestration and execution

[0078] (1) Visual process design

[0079] It provides a drag-and-drop workflow orchestration interface, supporting the addition of nodes such as data collection, preprocessing, quality inspection rule execution, and result output, and configuring the dependencies and data flow between nodes.

[0080] It supports control flows such as conditional branching (e.g., "If the quality inspection fails, proceed to the manual review process"), parallel processing (e.g., perform multi-dimensional quality inspections simultaneously), and loop execution (e.g., perform repeated quality inspections on batches of data).

[0081] (2) Dynamic process optimization

[0082] Process adjustments based on quality inspection results: If a certain type of data frequently triggers the same quality inspection rule to fail, the system will automatically adjust the process, adding a pre-verification node or increasing the execution priority of the rule.

[0083] Dynamic resource scheduling: Automatically adjusts the number of concurrent threads and allocates computing resources (such as CPU and memory) based on the workload of quality inspection tasks, and supports integration with Kubernetes clusters to achieve elastic scaling.

[0084] (3) Process execution and monitoring

[0085] Task scheduling engine: Supports three modes: scheduled scheduling (such as performing full quality inspection every morning), event-triggered (such as automatically triggering incremental quality inspection after data update), and manual triggering.

[0086] Real-time monitoring dashboard: Displays indicators such as process execution status, node time, quality inspection pass rate, and abnormal data distribution. It supports setting alarm thresholds (such as sending SMS or email notifications when quality inspection takes more than 30 minutes).

[0087] (V) Analysis and Feedback Optimization of Quality Inspection Results

[0088] (1) Multi-dimensional result visualization

[0089] Generates quality inspection reports, including data quality scores, statistics on the execution results of each rule, details of abnormal data (such as fields, error types, and frequency of occurrence), and trend analysis. Supports interactive filtering (such as filtering abnormal data by business line, data type, and error level) to facilitate locating the root cause of problems.

[0090] (2) Feedback-driven model iteration

[0091] Manual annotation and feedback: Provides an interface for annotating abnormal data, allowing users to mark false positives and false negatives in the large model. Annotated data is automatically used for fine-tuning the large model, improving the accuracy of subsequent quality inspections. Hint word optimization: Automatically adjusts the large model's hint words based on feedback results. For example, if a rule generation error is found to have omitted a certain type of business logic, the system automatically adds relevant constraints to the hint words.

[0092] (3) Abnormal data processing link

[0093] For abnormal data discovered during quality inspection, three processing methods are supported: automatic error correction (such as filling in missing values ​​and correcting format errors), marking as pending processing, and triggering downstream processes (such as pushing abnormal orders to manual review). The processing rules can be customized through process orchestration.

[0094] (vi) Implementation of key technology components

[0095] (1) Large Model Integration Component (LLMConnector)

[0096] It encapsulates mainstream large-scale model APIs (such as OpenAI GPT-4 and Anthropic Claude), supporting configurations such as model switching, temperature parameter adjustment, and maximum token limit. It also implements prompt word template management and historical dialogue record caching to reduce duplicate request overhead and improve the efficiency of large-scale model calls.

[0097] (2) FlowOrchestrator

[0098] The system uses a state machine to schedule and manage the status of process nodes, supporting process version control (such as rolling back to historical versions) and breakpoint resume (resuming execution from the last successful node after an abnormal interruption). It provides a RESTful API interface to support external system calls to trigger quality inspection processes and query process status.

[0099] (3) Rule Engine

[0100] It supports multiple rule execution methods, including SQL expressions, regular expressions, and custom scripts (Python / Java), and is compatible with traditional quality inspection rules and large model generation rules.

[0101] Implement rule priority management and conflict detection (such as automatic alarm when two rules have contradictory validation conditions for the same field).

[0102] (vii) Typical application scenario process examples

[0103] Taking e-commerce order data quality inspection as an example, the complete process is as follows:

[0104] 1. Data Acquisition: Order table data is incrementally collected every day at midnight using the QueryDatabaseByCursor component and stored in partitions by date.

[0105] 2. Preprocessing: Use DataFormatter to clean up null values, and TextSplitter to split the order remarks text into short sentences.

[0106] 3. Rule generation: The large model generates basic rules based on the order table structure (such as unique order number and amount > 0), and generates business rules by combining e-commerce business knowledge (such as "cancelled orders cannot have payment records").

[0107] 4. Parallel quality inspection:

[0108] Structured field validation: Performs non-null, value range, and uniqueness checks through the rule engine; Text semantic analysis: Analyzes sensitive information (such as user mobile phone numbers) and complaint keywords in the large model's notes text; Cross-table joins: Verifies whether the order user ID exists in the user table and whether the order product ID exists in the product table.

[0109] 5. Results Summary: Merge the quality inspection results from all dimensions, generate an order quality score, and mark high-risk orders (e.g., abnormal amount + remarks containing complaint terms).

[0110] 6. Anomaly Handling: High-risk orders are automatically pushed to customer service for review, and low-risk abnormal data is automatically corrected (such as filling in missing order statuses).

[0111] 7. Feedback Optimization: Customer service review results are fed back to the system to help fine-tune the business rule generation capabilities of the large model.

[0112] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An automated data quality inspection system based on large models and data flow orchestration, characterized by: include: Data access layer: Supports the acquisition of multi-source heterogeneous data, covering relational databases, unstructured data, streaming data, and big data, and enables flexible expansion of data sources through custom adapter components; Intelligent quality inspection layer: It consists of a large model inference engine, a rule engine, and a quality inspection algorithm library. The large model is responsible for rule generation and semantic understanding, the rule engine performs structured data verification, and the quality inspection algorithm library provides basic algorithms for anomaly detection and consistency verification. Workflow orchestration layer: Based on a low-code platform, it realizes visual orchestration of quality inspection processes, supports conditional branching, parallel processing, task scheduling, and can dynamically adjust process nodes according to quality inspection results; The intelligent quality inspection process driven by a large model in the intelligent quality inspection layer includes: Dynamic generation of quality inspection rules: Built-in common quality inspection rule templates and support for user-defined template extensions; guides large models to generate quality inspection rules for specific business scenarios through the Dify platform or custom prompt words; after generating rules, the effectiveness of the rules is tested through small-scale data, and the rule coverage and false alarm rate are automatically evaluated; users can manually adjust prompt words to optimize the rule generation effect. Semantic quality inspection of unstructured data: Using large models to perform semantic analysis on text data to identify issues such as sensitive information leakage, semantic ambiguity, and abnormal sentiment; for system logs, using large models to understand log semantics and detect abnormal operation records and compliance issues; Intelligent validation of structured data: Based on a large model to understand the relationships between tables, it automatically generates cross-table association rules; for anomaly detection of numerical fields, the large model dynamically calculates reasonable thresholds based on historical data distribution. The low-code process orchestration and execution process of the process orchestration layer includes: Visual workflow design: Provides a drag-and-drop workflow orchestration interface, supports adding data acquisition, preprocessing, quality inspection rule execution, and result output nodes, and configuring dependencies and data flow between nodes; supports conditional branching, parallel processing, and loop execution control flow; Dynamic process optimization: Automatically adjusts the process based on quality inspection results, adding pre-verification nodes or increasing the execution priority of rules; automatically adjusts the number of concurrent threads and allocates computing resources according to the load of quality inspection tasks, and supports integration with Kubernetes clusters to achieve elastic scaling. Process execution and monitoring: The task scheduling engine supports three modes: scheduled, event-triggered, and manual-triggered. The real-time monitoring dashboard displays the process execution status, node time, quality inspection pass rate, and abnormal data distribution indicators, and supports setting alarm thresholds.

2. The automated data quality inspection system based on large model and data flow orchestration according to claim 1, characterized in that: The data access and preprocessing process at the data access layer includes: Multi-source data acquisition: Use the custom extended DataSourceAdapter component to configure the data source type, connection parameters, and acquisition frequency; for database acquisition, the QueryDatabaseByCursor component is provided to support full, incremental, and full + incremental modes, and the data source, table, and incremental related parameters can be configured; Data preprocessing: The DataFormatter component is used for format standardization, including null value filling, format conversion, and special character filtering; the TextSplitter component is used to split unstructured text data by paragraph, sentence, or keyword, and to split structured data by row, column, or group it according to specified rules; data field names, types, lengths, and enumeration values ​​are automatically extracted to generate a data dictionary.

3. The automated data quality inspection system based on large model and data flow orchestration according to claim 2, characterized in that: It also includes the process of quality inspection result analysis and feedback optimization, specifically: Multi-dimensional result visualization: Generates quality inspection reports, including data quality scores, statistics of each rule's execution results, details of abnormal data, and trend analysis; supports interactive filtering of abnormal data. Feedback-driven model iteration: Provides an interface for anomaly data annotation, allowing users to mark false positives and false negatives in the large model. The annotated data is automatically used for fine-tuning the large model; the prompt words of the large model are automatically adjusted based on feedback results. Abnormal data processing chain: For abnormal data discovered by quality inspection, three processing methods are supported: automatic error correction, marking for processing, and triggering downstream processes. The processing rules can be customized through process orchestration.

4. An automated data quality inspection method based on large models and data flow orchestration, characterized in that: The system described in claim 3 is used in the following steps: System architecture: A three-tier architecture is adopted, including a data access layer, an intelligent quality inspection layer, and a process orchestration layer; The data access layer supports the acquisition of multi-source heterogeneous data and enables flexible expansion of data sources through custom adapter components; the intelligent quality inspection layer consists of a large model inference engine, a rule engine, and a quality inspection algorithm library; the process orchestration layer realizes the visual orchestration of the quality inspection process based on a low-code platform, supports conditional branching, parallel processing, and task scheduling functions, and can dynamically adjust process nodes according to the quality inspection results.

Images