Data cleaning conversion method and system based on large language model

Abstract

The invention relates to the technical field of data processing, in particular to a data cleaning and conversion method and system based on a large language model, and the method comprises the following steps: data input, data processing, data statistical analysis, manual auditing, rule adjustment and feedback, and data output. The method has the beneficial effects that data exploration and cleaning are carried out by integrating a large language model, and a dynamic and intelligent data cleaning process is realized by combining manual auditing and a rule feedback mechanism. The system supports the access of various data sources, has automatic processing capability, and ensures the accuracy and reliability of data processing at the same time. Besides, the system has good flexibility and expandability, can adapt to different application scenes, and improves the user experience and the transparency of data processing through a detailed processing report and a visual user interface.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data cleaning and conversion method and system based on a large language model. Background Art

[0002] In the era of big data, data has become the key cornerstone of decision-making and innovation in various fields. As an important prerequisite for data analysis and mining, data cleaning and conversion are becoming increasingly important. They ensure the quality and availability of data and provide a solid foundation for subsequent data analysis and model training.

[0003] Traditional data cleaning and conversion methods mainly rely on manually set rules and simple automated tool assistance. However, with the rapid development of the times, data sources have become more diverse than ever before, and the types and formats of data have become extremely complex. Especially when faced with unstructured data, the limitations of traditional methods are more obvious. The explosive growth of data volume has also posed a severe test to traditional data cleaning methods. Summary of the invention

[0004] The purpose of the present invention is to provide a data cleaning and conversion method and system based on a large language model to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a data cleaning and conversion method based on a large language model, the method comprising the following steps:

[0006] Data input, receiving raw data through the data input module and connecting the raw data to the system;

[0007] Data processing: Use the large language model to conduct preliminary exploration of the original data and process the data according to the initially set cleaning and conversion rules;

[0008] Data statistical analysis: compare the data before and after the large language model is cleaned and converted, and provide an analysis report on data processing;

[0009] Manual review: Manual review and analysis reports, and adjustment of some data usage rules based on the review results;

[0010] Rule adjustment and feedback: Based on the results of manual review, fine-tune the data cleaning conversion rules and feed the updated rules back to the large language model. The system automatically updates the data cleaning conversion rules.

[0011] Data output: When the data cleaning transformation rules no longer need to be adjusted, the final processed data is saved and a detailed processing report is generated.

[0012] Preferably, data input supports multiple types of data sources, such as file upload, database import, and API interface call methods; and performs preliminary preprocessing on the imported data, such as data decompression and data format conversion.

[0013] Preferably, data processing converts natural language data into a structured form by identifying and processing it for further data cleaning and analysis; cleaning rules include missing value processing, outlier processing, duplicate data processing, and error format processing; conversion rules include text to structured data, data encoding conversion, and measurement unit unification.

[0014] Preferably, data statistical analysis tracks the changing trend of data quality and provides a visual display so that users can better understand the effect of data processing;

[0015] Manual review accesses comparative information before and after data cleaning and conversion through a graphical user interface and submits corrections to ensure the accuracy and effectiveness of data processing.

[0016] Preferably, rule adjustment and feedback automatically updates the data cleaning conversion rules based on the feedback from the auditors, and records the specific content of each update for traceability and auditing; the data cleaning conversion rules automatically select the appropriate processing algorithm based on the data content, and are constantly improved with the participation of manual review;

[0017] Data output supports exporting processed data as files such as CSV, Excel, and JSON, exporting to a database, and calling an API to transmit results.

[0018] A data cleaning and conversion system based on a large language model is applied to a data cleaning and conversion method based on a large language model, and the system includes:

[0019] A data input module receives raw data through the data input module and connects the raw data to the system;

[0020] The data processing module uses a large language model to conduct a preliminary exploration of the original data and processes the data according to the initially set cleaning and conversion rules;

[0021] The data statistical analysis module is used to compare the data before and after the large language model is cleaned and converted, and to provide an analysis report on the data processing;

[0022] The manual review module is used to manually review the analysis report and adjust the rules for using some data based on the review results;

[0023] The rule adjustment and feedback module fine-tunes the data cleaning conversion rules based on the results of manual review, and feeds the updated rules back to the large language model. The system automatically updates the data cleaning conversion rules.

[0024] The data output module is used to save the final processed data and generate a detailed processing report when the data cleaning transformation rules no longer need to be adjusted.

[0025] Preferably, the data input module supports multiple types of data sources, such as file upload, database import, and API interface call methods; and performs preliminary preprocessing on the imported data, such as data decompression and data format conversion.

[0026] Preferably, the data processing module converts natural language data into a structured form by identifying and processing it for further data cleaning and analysis; the cleaning rules include missing value processing, outlier processing, duplicate data processing, and error format processing; the conversion rules include text to structured data, data encoding conversion, and measurement unit unification.

[0027] Preferably, the data statistical analysis module tracks the changing trend of data quality and provides a visual display so that users can better understand the effect of data processing;

[0028] The manual review module accesses the comparative information before and after data cleaning and conversion through a graphical user interface and submits corrections to ensure the accuracy and effectiveness of data processing.

[0029] Preferably, the rule adjustment and feedback module automatically updates the data cleaning conversion rules based on the feedback from the auditors, and records the specific content of each update for traceability and auditing; the data cleaning conversion rules automatically select the appropriate processing algorithm based on the data content, and are constantly improved with the participation of manual review;

[0030] The data output module supports exporting processed data to files such as CSV, Excel, and JSON, exporting to a database, and calling an API to transmit the results.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The data cleaning and conversion method and system based on the large language model proposed in the present invention realizes a dynamic and intelligent data cleaning process by integrating the large language model for data exploration and cleaning, and combining manual review and rule feedback mechanism. The system supports access to multiple data sources and has automatic processing capabilities, while ensuring the accuracy and reliability of data processing. In addition, the system has good flexibility and scalability, can adapt to different application scenarios, and improves the user experience and transparency of data processing through detailed processing reports and intuitive user interfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2 It is a flow chart of the data input module of the present invention;

[0035] Figure 3 This is a structural diagram of the data processing module of the present invention;

[0036] Figure 4 This is a flow chart for updating the large language model cleaning conversion rules of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] For example, see Figures 1 to 4 The present invention provides a technical solution: a data cleaning and conversion method based on a large language model, the method comprising the following steps:

[0039] 1. Data Input

[0040] Build a data input module to support access to multiple types of data sources. When there is a need to upload files, set up a file format verification mechanism to ensure that the uploaded files meet the system's required formats, such as CSV, Excel, JSON, etc. For database import, establish connection interfaces with different database types to be able to read data from common databases. In terms of API interface calls, configure security authentication and data transmission protocols to ensure stable data transmission.

[0041] Once the data is accessed, it is immediately pre-processed. If it is a compressed file, it is decompressed using an appropriate decompression algorithm to ensure that the file content is intact and undamaged.

[0042] 2. Data Processing

[0043] Build a data processing module and introduce a large language model. Through training on a large amount of data, the large language model can accurately identify different types of data features and form initial cleaning and conversion rules.

[0044] For missing value processing, methods such as mean filling, median filling or mode filling can be used according to the data type. In terms of outlier processing, the outlier range is determined by statistical analysis methods, and outliers that exceed the range are marked or replaced. For duplicate data, key features of the data are compared to identify and delete them. In error format processing, a recognition mode for common error formats is established to automatically correct the error format.

[0045] In terms of conversion rules, when converting text data to structured data, the semantic analysis capabilities of the large language model are used to extract key information and convert it into a structured table or database format. For situations where measurement units are not unified, a unit conversion table is established to automatically convert units.

[0046] 3. Data Statistical Analysis

[0047] Develop a data statistics analysis module to compare the data before and after the large language model is processed in detail. Calculate the changes in various data indicators, such as the mean, standard deviation, median, etc. of the data, and generate an analysis report on the data processing.

[0048] Establish a data quality change trend tracking mechanism to record the quality changes of data in real time during the cleaning and conversion process. Use visualization tools such as line charts and bar charts to intuitively display the data quality change trend so that users can better understand the effect of data processing.

[0049] 4. Manual Review

[0050] Design a manual review module and provide a graphical user interface for reviewers to access. The interface clearly displays the comparison information before and after data cleaning and conversion, including the original data, processed data, and changes in data indicators.

[0051] The auditor carefully checks the comparison information, marks the questionable data, and submits correction opinions. The correction opinions should explain the problem in detail and the recommended treatment method so that the system can better handle the subsequent processing.

[0052] 5. Rule Adjustment and Feedback

[0053] Build a rule adjustment and feedback module to fine-tune the data cleaning transformation rules based on the results of manual review. If the reviewer finds that the processing method for certain types of missing values ​​is inaccurate, the filling strategy can be adjusted. The processing range for outliers can also be adjusted according to actual conditions.

[0054] The updated rules are fed back to the large language model so that the model can process the next round of data according to the new rules. At the same time, the system automatically records the specific content of each rule update, including the reason, time, and data type involved, for subsequent tracing and auditing.

[0055] 6. Data Output

[0056] Set up a data output module to save the final processed data when the data cleaning conversion rules no longer need to be adjusted. Generate a detailed processing report, which should include the entire process of data processing, the application of various rules, and the evaluation results of data quality.

[0057] Provides multiple data output methods. The processed data can be exported to common file formats, such as CSV, Excel, JSON, etc., which is convenient for users to conduct further analysis in different software. Data can also be exported to a database to achieve data sharing with other systems. The processing results can also be transmitted to other applications through the API interface to improve the efficiency of data utilization.

[0058] Embodiment 2, based on embodiment 1, proposes a data cleaning and conversion system based on a large language model, which is applied to a data cleaning and conversion method based on a large language model. The system includes:

[0059] (a) Data input module: The data input module receives raw data and connects the raw data to the system.

[0060] Supports multiple types of data sources, such as file upload, database import, API interface call, etc.

[0061] Perform preliminary preprocessing on the imported data, such as data decompression, data format conversion, etc.

[0062] (b) Data processing module: Use a large language model to conduct a preliminary exploration of the raw data and process the data according to the initially set cleaning conversion rules.

[0063] Identify and process natural language data and convert it into a structured form for further data cleaning and analysis.

[0064] Cleaning rules can include missing value processing, outlier processing, duplicate data processing, and error format processing.

[0065] Conversion rules may include text to structured data, data encoding conversion, and measurement unit unification.

[0066] (c) Data statistical analysis module: compares the data before and after the large language model is cleaned and converted, and provides an analysis report on the data processing.

[0067] Track the changing trends of data quality and provide visual display so that users can better understand the effects of data processing.

[0068] (d) Manual review module: Manually review the analysis report and adjust the rules for using some data based on the review results.

[0069] Access comparison information before and after data cleaning and conversion through a graphical user interface and submit corrections to ensure the accuracy and effectiveness of data processing.

[0070] (e) Rule adjustment and feedback module: Based on the results of manual review, the data cleaning conversion rules are fine-tuned and the updated rules are fed back to the large language model. The system can automatically update the data cleaning conversion rules.

[0071] Ability to automatically update data cleansing transformation rules based on auditor feedback and record the specific content of each update for traceability and auditing.

[0072] Data cleaning transformation rules can automatically select appropriate processing algorithms based on data content, and are constantly being improved with the participation of manual review.

[0073] (f) Data output module: When the data cleaning transformation rules no longer need to be adjusted, the final processed data is saved and a detailed processing report is generated.

[0074] Supports exporting processed data to files such as CSV, Excel, JSON, etc., exporting to databases, calling APIs to transmit results, etc.

[0075] The present invention achieves efficient and accurate data cleaning conversion by introducing a large language model for data exploration and processing, combined with manual review and rule fine-tuning. This method not only improves the automation of data cleaning, but also ensures the accuracy and flexibility of data processing, providing a solid foundation for data analysis and mining.

[0076] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data cleaning and conversion method based on a large language model, characterized by: The method comprises the following steps: Data input, receiving raw data through the data input module and connecting the raw data to the system; Data processing: Use the large language model to conduct preliminary exploration of the original data and process the data according to the initially set cleaning and conversion rules; Data statistical analysis: compare the data before and after the large language model is cleaned and converted, and provide an analysis report on data processing; Manual review: Manual review of analysis reports, and adjustment of some data usage rules based on the review results; Rule adjustment and feedback: Based on the results of manual review, fine-tune the data cleaning conversion rules and feed the updated rules back to the large language model. The system automatically updates the data cleaning conversion rules. Data output: When the data cleaning transformation rules no longer need to be adjusted, the final processed data is saved and a detailed processing report is generated.

2. The data cleaning and conversion method based on a large language model according to claim 1, characterized in that: Data input supports multiple types of data sources, such as file upload, database import, and API interface call methods; it performs preliminary preprocessing on the imported data, such as data decompression and data format conversion.

3. The data cleaning and conversion method based on a large language model according to claim 1, characterized in that: Data processing identifies and processes natural language data and converts it into a structured form for further data cleaning and analysis; cleaning rules include missing value processing, outlier processing, duplicate data processing, and error format processing; conversion rules include text to structured data, data encoding conversion, and measurement unit unification.

4. The data cleaning and conversion method based on a large language model according to claim 1, characterized in that: Data statistical analysis tracks the changing trends of data quality and provides visual display so that users can better understand the effects of data processing; Manual review accesses comparative information before and after data cleaning and conversion through a graphical user interface and submits corrections to ensure the accuracy and effectiveness of data processing.

5. The data cleaning and conversion method based on a large language model according to claim 1, characterized in that: Rule adjustment and feedback automatically updates the data cleaning conversion rules based on the feedback from the auditors, and records the specific content of each update for traceability and auditing; the data cleaning conversion rules automatically select the appropriate processing algorithm based on the data content, and are constantly being improved with the participation of manual review; Data output supports exporting processed data as files such as CSV, Excel, and JSON, exporting to a database, and calling an API to transmit results.

6. A data cleaning and conversion system based on a large language model, applied to the data cleaning and conversion method based on a large language model according to any one of claims 1 to 5, characterized in that: The system comprises: A data input module receives raw data through the data input module and connects the raw data to the system; The data processing module uses a large language model to conduct a preliminary exploration of the original data and processes the data according to the initially set cleaning and conversion rules; The data statistical analysis module is used to compare the data before and after the large language model is cleaned and converted, and to provide an analysis report on the data processing; The manual review module is used to manually review the analysis report and adjust the rules for using some data based on the review results; The rule adjustment and feedback module fine-tunes the data cleaning conversion rules based on the results of manual review, and feeds the updated rules back to the large language model. The system automatically updates the data cleaning conversion rules. The data output module is used to save the final processed data and generate a detailed processing report when the data cleaning transformation rules no longer need to be adjusted.

7. The data cleaning and conversion system based on a large language model according to claim 6, characterized in that: The data input module supports various types of data sources, such as file upload, database import, and API interface call methods; it performs preliminary preprocessing on the imported data, such as data decompression and data format conversion.

8. The data cleaning and conversion system based on a large language model according to claim 6, characterized in that: The data processing module identifies and processes natural language data and converts it into a structured form for further data cleaning and analysis; the cleaning rules include missing value processing, outlier processing, duplicate data processing, and error format processing; the conversion rules include text to structured data, data encoding conversion, and measurement unit unification.

9. The data cleaning and conversion system based on a large language model according to claim 6, characterized in that: The data statistics analysis module tracks the changing trend of data quality and provides a visual display so that users can better understand the effect of data processing; The manual review module accesses the comparative information before and after data cleaning and conversion through a graphical user interface and submits corrections to ensure the accuracy and effectiveness of data processing.

10. The data cleaning and conversion system based on a large language model according to claim 6, characterized in that: The rule adjustment and feedback module automatically updates the data cleaning conversion rules based on the feedback from the auditors, and records the specific content of each update for traceability and auditing. The data cleaning conversion rules automatically select the appropriate processing algorithm based on the data content, and are constantly being improved with the participation of manual review. The data output module supports exporting processed data as files such as CSV, Excel, and JSON, exporting to a database, and calling an API to transmit the results.

Images