A large model-based data quality evaluation and cleaning system

Abstract

The application relates to the technical field of data governance, and discloses a data quality evaluation and cleaning system based on a large model. An access unit accesses data, a processing unit normalizes the accessed data and generates a data portrait, an evaluation unit receives the data portrait and performs rule evaluation and semantic evaluation, a scoring unit calculates a quality evaluation value PG, a center unit analyzes the quality evaluation value PG and calls a cleaning unit to clean the data, and an optimization unit optimizes the cleaning unit. The evaluation unit calls a large model to perform semantic understanding on the data portrait content, and predefines an entity inconsistency, context contradiction, abnormal value, semantic ambiguity and other problem type library. The evaluation unit can automatically count the number of the above semantic problems and calculate the proportion of the semantic problems, thereby making up for the deficiency that traditional rule evaluation can only identify surface problems, expanding data quality evaluation from a structure level to a semantic level, and significantly improving the comprehensiveness and accuracy of the evaluation result.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and more specifically to a data quality assessment and cleaning system based on a large model. Background Technology

[0002] Against the backdrop of accelerated digital transformation, data has become a key production factor for enterprise and organizational operations and management. With the explosive growth of data scale and the increasing diversification of data sources, data quality issues are becoming increasingly prominent, such as data missing, inconsistent formats, duplicate primary keys, semantic ambiguity, and entity conflicts. These problems frequently occur, seriously restricting the accuracy of data-driven decision-making and the stability of system operation. High-quality data is the foundation for data analysis, artificial intelligence applications, and business process automation. Therefore, building an efficient and automated data quality assessment and cleaning mechanism has become one of the core challenges in the field of data governance. Traditional data quality assessment methods mainly rely on manually predefined rules such as completeness, uniqueness, and format compliance to verify data. These methods can only identify obvious structural errors, and their assessment results are not comprehensive or accurate enough. Traditional methods lack a unified and standardized processing mechanism when accessing data, which leads to inconsistencies in the data foundation for subsequent assessment and cleaning work, thus affecting subsequent processing. Existing data cleaning systems usually use fixed rules or manually written scripts to repair data. The cleaning strategy cannot be self-optimized based on historical cleaning results, nor can it quickly reuse proven cleaning solutions in new problem scenarios. Summary of the Invention

[0003] To overcome the aforementioned deficiencies of the prior art, the embodiments of the present invention provide a data quality assessment and cleaning system based on a large model to solve the technical problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a data quality assessment and cleaning system based on a large model, comprising an access unit, a processing unit, an evaluation unit, a scoring unit, a central unit, a cleaning unit, and an optimization unit. The access unit is used to access data; the processing unit performs normalization processing and data profile generation on the accessed data; the evaluation unit receives the data profile and performs rule evaluation and semantic evaluation; the scoring unit is used to calculate the quality assessment value PG; the central unit analyzes the quality assessment value PG and calls the cleaning unit to perform data cleaning; the cleaning unit performs data cleaning; and the optimization unit optimizes the cleaning unit. The access unit dynamically loads the corresponding driver and parsing strategy from the adapter registry based on the data source configuration information through the adapter. After the connection is established, the access unit calls the metadata interface of each data source to obtain a complete data structure description. The access unit identifies the range of data to be acquired according to the configured access strategy and collects the raw data of each data source in batch streaming mode. The access unit sends the received data to the processing unit.

[0005] In a preferred embodiment, the processing unit receives data from the access unit and performs normalization processing and data profile generation. The processing unit performs character encoding unification, data type alignment, null value marking, and unit normalization processing to transform the received data into unified data. The unified data has unified field types, unified encoding format, unified null value representation, unified measurement unit, and consistent data structure.

[0006] In a preferred embodiment, the processing unit performs multi-dimensional statistical analysis on the unified data. The multi-dimensional statistical analysis includes field-level statistics on null value rate, number of unique values, numerical distribution characteristics, and string length distribution; table-level statistics on total number of records, data size, and primary key duplication rate; and relationship-level analysis on foreign key associations and field name similarity to identify cross-table relationships and generate a data profile. The processing unit then sends the generated data profile to the evaluation unit.

[0007] In a preferred embodiment, the evaluation unit receives the data profile sent by the processing unit, and the evaluation unit includes a rule evaluation module and a semantic evaluation module. The rule evaluation module includes four rules: uniqueness rule, integrity rule, format rule, and range rule. The rule evaluation module runs the four rules to evaluate four proportion data: primary key duplication rate (CD), null value rate (ZB), format error rate (GS), and out-of-range rate (FW). The evaluation unit sends the calculated four proportion data to the scoring unit.

[0008] In a preferred embodiment, the semantic evaluation module performs semantic understanding of the data profile content by calling a large model. The large model called by the semantic evaluation module predefines a library of entity inconsistencies, contextual contradictions, outliers, and semantic ambiguities. The semantic evaluation module counts the number of entity inconsistencies, contextual contradictions, outliers, and semantic ambiguities, calculates the ratio of this number to the total amount of data, calculates the semantic problem proportion YY, and sends the calculated semantic problem proportion YY to the scoring unit.

[0009] In a preferred embodiment, the scoring unit receives four proportion data from the rule evaluation module and the semantic question proportion YY sent by the semantic evaluation module, and calculates a quality evaluation value PG. The formula for calculating the quality evaluation value PG is as follows: In the formula, i represents the current i-th type, ki represents the weights corresponding to all five data received by the scoring unit, and ZBi represents the five data received by the scoring unit. The scoring unit sends the calculated quality assessment value PG to the central unit.

[0010] In a preferred embodiment, the central unit receives a quality assessment value PG and compares it with its internal quality threshold ZY. When the quality assessment value PG ≥ the quality threshold ZY, the central unit does not perform data processing. When the quality assessment value PG < the instruction threshold ZY, the central unit determines that the data has a data problem. The central unit will call the cleaning unit to perform data cleaning. The data includes rule problems counted by the rule evaluation module and semantic problems understood by the semantic evaluation module.

[0011] In a preferred embodiment, when the cleaning unit performs data cleaning, it matches and retrieves the rule questions in the strategy knowledge base. The strategy knowledge base stores historical cleaning strategies in the form of key-value pairs and vector databases. Each strategy includes the applicable question type, field features, business scenario tags, and specific cleaning operation expressions. By calculating the similarity between the current question features and the strategies in the knowledge base, the verified and effective cleaning schemes are reused for cleaning.

[0012] In a preferred embodiment, the cleaning unit calls a large model to perform in-depth analysis of the semantic problem, combines the problem description, field metadata, data profile information, and business scenario description into structured prompt words and inputs them into the large model. Based on its semantic understanding capabilities, the large model outputs cleaning suggestions in natural language form, simulates the execution of the generated cleaning strategy, and performs the cleaning operation after ensuring the feasibility of the strategy.

[0013] In a preferred embodiment, the optimization unit records each cleaning operation of the cleaning unit and the feedback results after execution. The optimization unit uses the feedback results to optimize and fine-tune the large model. The optimization module stores the confirmed effective cleaning strategies and applicable scenario conditions into the strategy knowledge base every day, and the strategy knowledge base is optimized and adjusted daily.

[0014] The technical effects and advantages of this invention are as follows: 1. The evaluation unit of this invention performs semantic understanding of the data profile content by calling a large model and predefines a problem type library such as entity inconsistency, contextual contradiction, outliers, and semantic ambiguity. It can automatically count the number of the above semantic problems and calculate the proportion of semantic problems, thereby making up for the shortcomings of traditional rule evaluation which can only identify surface problems. It expands data quality evaluation from the structural level to the semantic level, significantly improving the comprehensiveness and accuracy of the evaluation results. 2. This invention sets up a unified and standardized processing flow through the processing unit, and standardizes and aligns character encoding, data type, null value marker, and unit, etc., to transform multi-source heterogeneous data into unified data with unified field type, unified encoding format, unified null value representation, unified measurement unit, and consistent data structure, generating a complete data profile, thereby solving the heterogeneity problem after multi-source data access and providing high-quality and standardized data input for the future. 3. This invention constructs a strategy knowledge base in the cleaning unit, stores historical cleaning strategies in the form of key-value pairs and vector databases, and reuses verified effective cleaning solutions to process rule-based problems by calculating the similarity between the current problem features and the strategies in the knowledge base. It calls a large model to output cleaning suggestions in natural language form and performs simulated execution verification. The optimization unit uses the feedback results to fine-tune and optimize the large model, thereby enabling the cleaning strategy to continuously iterate with changes in business scenarios and improving the intelligence level of cleaning operations. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall system composition of the present invention. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The data quality assessment and cleaning system based on a large model involved in the present invention is not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Reference Figure 1This invention provides a data quality assessment and cleaning system based on a large model, comprising an access unit, a processing unit, an evaluation unit, a scoring unit, a central unit, a cleaning unit, and an optimization unit. The access unit is used to access data. The processing unit performs normalization processing and data profiling on the accessed data. The evaluation unit receives the data profiling and performs rule evaluation and semantic evaluation. The scoring unit is used to calculate the quality assessment value PG. The central unit analyzes the quality assessment value PG and calls the cleaning unit to perform data cleaning. The cleaning unit performs the data cleaning work. The optimization unit optimizes the cleaning unit.

[0018] In this embodiment, the access unit uses an adapter mechanism to dynamically load drivers and parsing strategies, supporting flexible access to multi-source heterogeneous data. The processing unit realizes data normalization and profile generation, providing standardized input for subsequent evaluation. The evaluation unit and scoring unit realize quality quantification evaluation combining rules and semantics. The central unit intelligently calls the cleaning unit to perform cleaning based on the evaluation results. The optimization unit continuously optimizes the cleaning strategy. Therefore, the various units in this application work together to realize full-process automation from data access, evaluation, cleaning to strategy optimization, thereby improving the intelligence level and overall efficiency of data governance.

[0019] Reference Figure 1 The access unit dynamically loads the corresponding driver and parsing strategy from the adapter registry based on the data source configuration information through the adapter. After the connection is established, the access unit calls the metadata interface of each data source to obtain a complete data structure description. The access unit identifies the range of data to be acquired according to the configured access strategy and collects the raw data of each data source in batch streaming mode. The access unit sends the received data to the processing unit.

[0020] In this embodiment, the access unit dynamically loads the corresponding driver and parsing strategy from the adapter registry based on the data source configuration information, thereby enabling flexible access to multi-source heterogeneous data. New data source types can be dynamically extended without modifying the system code, significantly improving scalability and adaptability. Furthermore, the access unit obtains a complete data structure description by calling the metadata interface of each data source, thereby accurately understanding the field definitions, type constraints, and other metadata of the data source, providing sufficient contextual support for subsequent standardization processing.

[0021] Reference Figure 1The processing unit receives data from the access unit and performs normalization processing and data profile generation. The processing unit performs character encoding unification, data type alignment, null value marking, and unit normalization processing to transform the received data into unified data. This unified data has unified field types, unified encoding formats, unified null value representation, unified measurement units, and consistent data structures. The processing unit will perform multi-dimensional statistical analysis on the unified data. The multi-dimensional statistical analysis includes field-level statistics on null value rate, number of unique values, numerical distribution characteristics, and string length distribution; table-level statistics on total number of records, data size, and primary key duplication rate; and relationship-level identification of cross-table relationships through foreign key associations and field name similarity to generate a data profile. The processing unit sends the generated data profile to the evaluation unit.

[0022] In this embodiment, the application transforms heterogeneous data from multiple sources into unified data with consistent field types, encoding formats, null value representations, measurement units, and data structures by performing character encoding unification, data type alignment, null value marking, and unit normalization. This solves the heterogeneity problem between different data sources, eliminates processing obstacles caused by data format differences, provides a standardized data input foundation for subsequent evaluation and cleaning processes, ensures efficient collaboration and processing consistency among system modules, and allows the processing unit to perform multi-dimensional statistical analysis on the unified data, covering the field level (null value rate, number of unique values, numerical distribution characteristics, string length distribution), the table level (total number of records, data size, primary key duplication rate), and the relationship level (foreign key associations, field name similarity identification of cross-table relationships), generating a complete data profile.

[0023] Reference Figure 1 The evaluation unit receives the data profile sent by the processing unit and includes a rule evaluation module and a semantic evaluation module. The rule evaluation module includes four rules: uniqueness rule, integrity rule, format rule, and range rule. The rule evaluation module runs the four rules to evaluate four proportions of primary key duplication (CD), null value proportion (ZB), format error proportion (GS), and out-of-range proportion (FW). The evaluation unit sends the calculated four proportions to the scoring unit. The semantic evaluation module performs semantic understanding of the data profile content by calling a large model. The large model called by the semantic evaluation module predefines a library of entity inconsistency, context contradiction, outlier, and semantic ambiguity problem types. The semantic evaluation module counts the number of entity inconsistency, context contradiction, outlier, and semantic ambiguity problems, calculates the ratio of this number to the total amount of data, and calculates the semantic problem proportion (YY). The semantic evaluation module sends the calculated semantic problem proportion (YY) to the scoring unit.

[0024] In this embodiment, the primary key duplication rate (CD) is the proportion of duplicate primary keys to the total data volume; the null value rate (ZB) is the proportion of missing null values ​​to the total data volume; the format error rate (GS) is the proportion of data with incorrect format to the total data volume; and the out-of-range rate (FW) is the proportion of data exceeding the limit. This application sets four core rules—uniqueness, integrity, format, and range—and calculates four key proportions for primary key duplication, null values, format errors, and out-of-range. This four-dimensional rule evaluation mechanism covers the most basic and critical dimensions of data quality: integrity, uniqueness, compliance, and accuracy. By quantifying proportion indicators, it intuitively reflects the degree of problems at the structural level, providing clear and quantifiable basic indicators for quality assessment. By calling a large model to perform semantic understanding of the data profile and predefining a library of entity inconsistencies, contextual contradictions, outliers, and semantic ambiguities, it automatically counts the number of semantic problems and calculates the proportion of semantic problems. This overcomes the limitation of traditional rule evaluation, which can only identify structural problems, thereby identifying and quantifying deep-seated semantic quality problems in the data, improving the comprehensiveness and intelligence of the evaluation.

[0025] Reference Figure 1 The scoring unit receives four proportion data from the rule evaluation module and the semantic question proportion YY sent by the semantic evaluation module, and calculates the quality evaluation value PG. The formula for calculating the quality evaluation value PG is as follows: In the formula, i represents the current i-th type, ki represents the weights corresponding to all five data points received by the scoring unit, and ZBi represents the five data points received by the scoring unit. The scoring unit sends the calculated quality assessment value PG to the central unit. The central unit receives the quality assessment value PG and compares it with its internal quality threshold ZY. When the quality assessment value PG ≥ the quality threshold ZY, the central unit does not perform data processing. When the quality assessment value PG < the instruction threshold ZY, the central unit determines that the data has a data problem. The central unit will call the cleaning unit to perform data cleaning. The data includes the rule problems counted by the rule evaluation module and the semantic problems understood by the semantic evaluation module.

[0026] In this embodiment, the proportions of four proportional data points and semantic issues are used to calculate a quality assessment value PG using a weighted summation formula. This scoring mechanism integrates structural quality issues and semantic quality issues into a unified quantitative evaluation system. Through configurable weight coefficients, it flexibly adapts to the differences in attention to various quality issues in different business scenarios, outputting an intuitive comprehensive quality score to provide a quantifiable basis for decision-making. The quality assessment value PG is compared with the quality threshold ZY. When the quality assessment value PG ≥ the quality threshold ZY, the data is deemed qualified and no additional processing is performed to avoid unnecessary computational overhead. When the quality assessment value PG < the instruction threshold ZY, the data is automatically determined to have quality issues, and the cleaning unit is precisely invoked to perform differentiated cleaning for rule-related and semantic issues. Therefore, this application can achieve on-demand cleaning, avoiding the inefficiency of full-volume cleaning, and optimizing resource utilization efficiency while ensuring data quality.

[0027] Reference Figure 1 When the cleaning unit performs data cleaning, it matches and retrieves the rule questions in the strategy knowledge base. The strategy knowledge base stores historical cleaning strategies in the form of key-value pairs and vector databases. Each strategy includes the applicable question type, field features, business scenario tags, and specific cleaning operation expressions. By calculating the similarity between the current question features and the strategies in the knowledge base, the cleaning unit reuses the verified and effective cleaning solutions for cleaning. The cleaning unit calls a large model to perform in-depth analysis of semantic questions, combines the question description, field metadata, data profile information, and business scenario description into structured prompt words and inputs them into the large model. Based on its semantic understanding capabilities, the large model outputs cleaning suggestions in natural language form. The generated cleaning strategy is simulated and executed to ensure the feasibility of the strategy before performing the cleaning operation.

[0028] In this embodiment, the application constructs a strategy knowledge base to store historical cleaning strategies in the form of key-value pairs and vector databases. Each strategy includes a problem type, field features, business scenario tags, and cleaning operation expressions. The similarity between the current problem features and the strategies in the knowledge base is calculated, and verified effective cleaning solutions are automatically reused to process rule-based problems, achieving intelligent reuse of cleaning strategies and avoiding repetitive manual configuration. For semantic problems, a large model is invoked for in-depth analysis. The problem description, field metadata, data profile information, and business scenario description are combined into structured prompt words and input into the large model to obtain cleaning suggestions in natural language form. The cleaning is then implemented after simulated execution verification. Through the semantic understanding and generation capabilities of the large model, the application can intelligently handle complex semantic problems such as entity inconsistency and contextual contradictions. At the same time, simulated execution ensures the feasibility of the strategy, balancing the intelligence and security of the cleaning process.

[0029] Reference Figure 1The optimization unit records each cleaning operation of the cleaning unit and the feedback results after execution. The optimization unit uses the feedback results to optimize and fine-tune the large model. The optimization module stores the confirmed effective cleaning strategies and applicable scenario conditions into the strategy knowledge base every day. The strategy knowledge base is optimized and adjusted daily.

[0030] In this embodiment, the application records each cleaning operation and feedback result, and uses the feedback result to optimize and fine-tune the large model. The effective cleaning strategies and applicable scenario conditions are stored in the strategy knowledge base every day. Therefore, the application can continuously learn and accumulate from actual cleaning operations. The semantic analysis capability of the large model is continuously enhanced with fine-tuning, and the strategy knowledge base is enriched with the accumulation of effective strategies, realizing the adaptive evolution of the data quality assessment and cleaning system.

[0031] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. The units and algorithm steps of the various examples described in the embodiments can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0032] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0033] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0034] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data quality assessment and cleaning system based on a large model, characterized in that: The system includes an access unit, a processing unit, an evaluation unit, a scoring unit, a central unit, a cleaning unit, and an optimization unit. The access unit is used to access data. The processing unit performs normalization processing on the accessed data and generates data profiles. The evaluation unit receives the data profiles and performs rule evaluation and semantic evaluation. The scoring unit is used to calculate the quality evaluation value PG. The central unit analyzes the quality evaluation value PG and calls the cleaning unit to perform data cleaning. The cleaning unit performs data cleaning. The optimization unit optimizes the cleaning unit. The access unit dynamically loads the corresponding driver and parsing strategy from the adapter registry based on the data source configuration information through the adapter. After the connection is established, the access unit calls the metadata interface of each data source to obtain a complete data structure description. The access unit identifies the range of data to be acquired according to the configured access strategy and collects the raw data of each data source in batch streaming mode. The access unit sends the received data to the processing unit.

2. The data quality assessment and cleaning system based on a large model according to claim 1, characterized in that: The processing unit receives data from the access unit and performs normalization processing and data profile generation. The processing unit performs character encoding unification, data type alignment, null value marking, and unit normalization processing to transform the received data into unified data. The unified data has unified field types, unified encoding format, unified null value representation, unified measurement unit, and consistent data structure.

3. The data quality assessment and cleaning system based on a large model according to claim 1, characterized in that: The processing unit performs multi-dimensional statistical analysis on the unified data. The multi-dimensional statistical analysis includes field-level statistics on null value rate, number of unique values, numerical distribution characteristics, and string length distribution; table-level statistics on total number of records, data size, and primary key duplication rate; and relationship-level statistics on cross-table relationships through foreign key associations and field name similarity to generate a data profile. The processing unit then sends the generated data profile to the evaluation unit.

4. The data quality assessment and cleaning system based on a large model according to claim 1, characterized in that: The evaluation unit receives the data profile sent by the processing unit, and the evaluation unit includes a rule evaluation module and a semantic evaluation module. The rule evaluation module includes four rules: uniqueness rule, integrity rule, format rule, and range rule. The rule evaluation module runs the four rules to evaluate four proportions of primary key duplication (CD), null value proportion (ZB), format error proportion (GS), and out-of-range proportion (FW). The evaluation unit sends the calculated four proportions of data to the scoring unit.

5. The data quality assessment and cleaning system based on a large model according to claim 4, characterized in that: The semantic evaluation module performs semantic understanding of the data profile content by calling a large model. The large model called by the semantic evaluation module predefines a library of entity inconsistencies, contextual contradictions, outliers, and semantic ambiguities. The semantic evaluation module counts the number of entity inconsistencies, contextual contradictions, outliers, and semantic ambiguities, calculates the ratio of this number to the total amount of data, and calculates the semantic problem proportion YY. The semantic evaluation module sends the calculated semantic problem proportion YY to the scoring unit.

6. The data quality assessment and cleaning system based on a large model according to claim 1, characterized in that: The scoring unit receives four proportion data points from the rule evaluation module and the semantic question proportion YY sent by the semantic evaluation module, and calculates the quality evaluation value PG. The formula for calculating the quality evaluation value PG is as follows: In the formula, i represents the current i-th type, ki represents the weights corresponding to all five data received by the scoring unit, and ZBi represents the five data received by the scoring unit. The scoring unit sends the calculated quality assessment value PG to the central unit.

7. The data quality assessment and cleaning system based on a large model according to claim 6, characterized in that: The central unit receives the quality assessment value PG and compares it with its internal quality threshold ZY. When the quality assessment value PG ≥ the quality threshold ZY, the central unit does not perform data processing. When the quality assessment value PG < the instruction threshold ZY, the central unit determines that the data has a data problem. The central unit will call the cleaning unit to perform data cleaning. The data includes the rule problems counted by the rule evaluation module and the semantic problems understood by the semantic evaluation module.

8. The data quality assessment and cleaning system based on a large model according to claim 1, characterized in that: When the cleaning unit performs data cleaning, it matches and retrieves the rule questions in the strategy knowledge base. The strategy knowledge base stores historical cleaning strategies in the form of key-value pairs and vector databases. Each strategy includes the applicable question type, field features, business scenario tags and specific cleaning operation expressions. By calculating the similarity between the current question features and the strategies in the knowledge base, the unit reuses the verified and effective cleaning schemes for cleaning.

9. A data quality assessment and cleaning system based on a large model according to claim 8, characterized in that: The cleaning unit calls the large model to perform in-depth analysis of semantic issues, combines the problem description, field metadata, data profile information, and business scenario description into structured prompt words and inputs them into the large model. Based on its semantic understanding capabilities, the large model outputs cleaning suggestions in natural language form, simulates the execution of the generated cleaning strategy, and performs the cleaning operation after ensuring the feasibility of the strategy.

10. A data quality assessment and cleaning system based on a large model according to claim 1, characterized in that: The optimization unit records each cleaning operation of the cleaning unit and the feedback results after execution. The optimization unit uses the feedback results to optimize and fine-tune the large model. The optimization module stores the confirmed effective cleaning strategies and applicable scenario conditions into the strategy knowledge base every day. The strategy knowledge base is optimized and adjusted daily.

Images