Data cleaning method and device based on large language model and medium

By combining large language models and knowledge bases to generate ordered execution instructions, the problem of cumbersome data cleaning processes and high error rates in existing technologies is solved, achieving efficient and accurate data cleaning and ensuring data quality and consistency.

CN121833690APending Publication Date: 2026-04-10HANGZHOU YUNSHEN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-01-26
Publication Date
2026-04-10

AI Technical Summary

Technical Problem

Existing technologies rely on manually written scripts for data cleaning, which is cumbersome, time-consuming, and prone to errors. Furthermore, the scripts output by large models may contain logical errors, affecting the quality and efficiency of data cleaning.

Method used

By combining a pre-set large language model and knowledge base, ordered execution instructions are generated, including semantic parsing, instruction orchestration, and task program scripts, ensuring the rationality and reliability of the instructions. Multiple knowledge bases are used to improve the reliability and logic of the task execution process.

Benefits of technology

It improved the quality and efficiency of data cleaning, reduced the error rate, ensured the accuracy and consistency of data cleaning, avoided the generation of dirty data, and improved the accuracy of data query and analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121833690A_ABST
    Figure CN121833690A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, in particular to a data cleaning method and device based on a large language model.The method comprises the steps that a content text and a processing task corresponding to a to-be-processed data table input by a user are received, and a user intention statement is generated through a preset large language model; calling a first preset knowledge base to guide a preset large language model, generating a plurality of disordered execution instructions corresponding to the processing task, generating an ordered execution instruction set based on a preset instruction arrangement rule and the preset large language model, combining and sending the ordered execution instruction set to a back end, receiving a script generation instruction sent by the back end, sending the execution instructions to the preset large language model to enable the preset large language model to generate task program scripts corresponding to the execution instructions, and enabling the back end to run the task program scripts in order to realize data cleaning and obtain execution results of the processing tasks; according to the method, the reasonability and reliability of task execution process generation are improved, and the data cleaning quality and efficiency are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data cleaning method, device and medium based on a large language model. Background Technology

[0002] In the field of data analytics and applications, enterprises typically need to access diverse data tables from numerous external or internal business systems. These raw data vary significantly in format, naming, encoding rules, and quality, and must undergo cleaning, mapping, verification, and standardization processes to be transformed into high-quality data assets that conform to local standards and can be uniformly queried and analyzed.

[0003] Currently, the industry commonly employs a combination of rule-based scripts and manual methods for data cleaning. Specifically, technicians need to pre-write numerous scripts or SQL statements, defining data mapping relationships and cleaning rules. When new data tables are added, the table structure must be manually analyzed, and new scripts adjusted or written to adapt. This approach is highly dependent on the experience of technicians, resulting in a cumbersome, time-consuming process with a high rate of error. Furthermore, predefined rule-based scripts lack flexibility and cannot adaptively generate data cleaning processes that precisely match the task intent. In recent years, the emergence of large-scale models has offered new possibilities for automated data processing; however, the output of large-scale models carries uncertainty and the risk of "illusions." Directly generated program scripts may contain logical errors, making reliability difficult to guarantee and affecting the quality of data cleaning. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a data cleaning method, device, and medium based on a large language model. By analyzing user intent and generating several ordered execution instructions under the guidance of a preset knowledge base, the rationality and reliability of task execution process generation are improved, ensuring the quality and efficiency of data cleaning.

[0005] According to a first aspect of the present invention, a data cleaning method based on a large language model is provided, comprising the following steps: S1 receives the content text and processing task corresponding to the data table to be processed input by the user, performs semantic parsing through a preset large language model, and generates the user intent statement.

[0006] S2, the constructed dataset to be processed is sent to the preset large language model, and the first preset knowledge base is invoked to guide the preset large language model to generate several unordered execution instructions corresponding to the processing task; the dataset to be processed includes the content text, field attribute information, processing task and user intent statement corresponding to the data table to be processed; the first preset knowledge base includes several preset instructions, execution rules and access protocols corresponding to each preset instruction.

[0007] S3 calls the preset instruction arrangement rules in the second preset knowledge base so that the preset large language model can sort and classify several unordered execution instructions according to the preset instruction arrangement rules, generate an ordered set of execution instructions and send it to the backend.

[0008] S4 receives script generation instructions sent by the backend to the preset large language model based on the set of execution instructions, so that the preset large language model determines the task type of each execution instruction, and generates the task program script corresponding to each execution instruction according to the task execution template corresponding to the task type of each execution instruction and the table structure information corresponding to the retrieved data table to be processed.

[0009] S5 sends each task program script to the backend, so that the backend can run each task program script in an orderly manner in combination with the actual data in the data table to be processed, so as to clean the data in the data table to be processed into standardized data and obtain the execution results of the processing task.

[0010] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the above-described data cleaning method based on a large language model.

[0011] According to a third aspect of the present invention, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0012] The present invention has at least the following beneficial effects: This invention provides a data cleaning method based on a large language model. First, it receives the content text corresponding to a data table to be processed and the processing task input by the user. Then, it performs semantic parsing using a preset large language model to generate the user's intent statement, providing a basis for generating subsequent execution instructions. Next, it sends the content text, field attribute information, processing task, and user intent statement corresponding to the data table to be processed to the preset large language model. A first preset knowledge base guides the preset large language model, generating several unordered execution instructions corresponding to the processing task. Finally, it invokes preset instruction arrangement rules from a second preset knowledge base, enabling the preset large language model to generate an ordered set of execution instructions based on these rules and send it to the backend. By introducing two knowledge bases with different functions, the task execution process is improved. The system ensures the reliability and logical rationality of the generated data. It then receives script generation instructions from the backend, based on the set of execution instructions, and sends them to a pre-defined large language model. This allows the pre-defined large language model to determine the task type of each execution instruction. Based on the task execution template corresponding to the task type of each instruction and the table structure information of the retrieved data table, a task program script is generated for each execution instruction. The generated program script is more accurate due to the set of execution instructions and the defined task execution template. Executing the program script allows data from different platforms to be automatically stored locally in the same format, reducing data access time and error rates. This ensures data cleaning quality and efficiency, avoids the generation of dirty data, and ultimately improves the accuracy and reliability of data query and data analysis results. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of a data cleaning method based on a large language model provided in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] This invention provides a data cleaning method based on a large language model, such as... Figure 1As shown, the method includes the following steps: S1 receives the content text and processing task corresponding to the data table to be processed from the user's input, performs semantic parsing through a preset large language model, and generates the user's intent statement; it can be understood that: the content text corresponding to the data table to be processed refers to the text that the user selects a part of the table sample from the data table to be processed and decomposes it into key-value pairs, that is, the field is used as the key and the field value is used as the value.

[0017] As described above, in order to facilitate users' data query and analysis, in this embodiment, the user cleans the data of the data table to be processed into standardized data and stores it in a preset platform for subsequent task execution. Before that, it is necessary to parse the structured and specific user intent based on the user's input text and processing task, such as the task of analyzing table data, etc. By generating user intent statements, a basis is provided for the generation of subsequent execution instructions.

[0018] S2, the constructed dataset to be processed is sent to the preset large language model, and the first preset knowledge base is invoked to guide the preset large language model to generate several unordered execution instructions corresponding to the processing task; the dataset to be processed includes the content text, field attribute information, processing task and user intent statement corresponding to the data table to be processed; the first preset knowledge base includes several preset instructions, the execution rules and access protocols corresponding to each preset instruction; it can be understood that: by analyzing the dataset to be processed, the preset large language model is made to find the preset instructions related to the processing task based on the task flow information stored in the first preset knowledge base, rather than the preset large language model itself giving the execution instructions.

[0019] Specifically, the plurality of unordered preset instructions include, but are not limited to, insight instructions, cleaning instructions, and verification instructions.

[0020] Specifically, the field attribute information includes, but is not limited to, the character length of the field, the field type, and the field semantics; for example, the field type can be text or numeric.

[0021] As mentioned above, when generating execution instructions, it does not rely solely on the large language model for automatic generation. Instead, under the guidance of the first preset knowledge base, certain restrictions are placed on the generation of execution instructions from the large language model. This ensures that the generated execution instructions meet the execution requirements of the current data cleaning and processing tasks. Furthermore, the first preset knowledge base also specifies execution rules and access protocols, avoiding situations where unclear access specifications and standards can lead to task confusion when there is a large amount of input table data. This is beneficial for improving the quality of data cleaning.

[0022] Specifically, step S2 also includes the following steps: S201, when the dataset to be processed is received, check whether each key field corresponding to the data table to be processed is missing; the key field refers to the field that is semantically related to the processing task as determined by the preset large language model. For example, the field related to statisticians is the ID number or passport, and the field related to a certain business is the project name.

[0023] S202, if any key fields are missing, provide feedback on the missing information to the user; otherwise, calculate the data saturation corresponding to each key field.

[0024] Specifically, the data saturation D for the key field is 1-K, where K is the null value rate for the key field.

[0025] S203, when the data saturation corresponding to each key field is greater than the preset saturation threshold, the first preset knowledge base is called to guide the preset large language model and generate several unordered execution instructions corresponding to the processing task; otherwise, correction information is fed back to the user.

[0026] As described above, before generating execution instructions, a preliminary data check is performed on the dataset to be processed to determine whether the data information meets the requirements of the processing task. If the check fails, feedback is given to the user so that the user can make timely modifications, ensuring the accuracy of the subsequent processing task results.

[0027] S3 calls the preset instruction arrangement rules in the second preset knowledge base so that the preset large language model can sort and classify several unordered execution instructions according to the preset instruction arrangement rules, generate an ordered set of execution instructions and send it to the backend.

[0028] Specifically, the instruction orchestration rules include grouping several execution instructions targeting the same data table into the same execution instruction set, and sorting the execution instructions within the execution instruction set according to preset operational dependencies between the execution instructions. For example, when cleaning data from two tables is required, two sets of unordered execution instructions are generated, distinguished by table names, into two types of tasks, and two execution instruction sets are generated according to preset instruction orchestration rules.

[0029] Preferably, the preset large language model used in steps S1-S3 is the first preset intelligent agent in the preset large language model, and the preset large language model used in steps S4-S5 is the second preset intelligent agent in the preset large language model; wherein, the first intelligent agent and the second intelligent agent are based on the same large language model instance, and achieve functional differentiation by loading different system prompt words and contextual knowledge.

[0030] As described above, after obtaining several unordered execution instructions related to the processing task from the dataset to be processed, the pre-set large language model needs to sort the several unordered execution instructions under the guidance of the pre-set instruction arrangement rules, so that the sorting result meets the pre-set arrangement requirements, ensuring that the task process is accurate and logical. Furthermore, two different knowledge bases are used for the generation and arrangement of execution instructions, so that each knowledge base can be trained according to the vertical domain corresponding to its own task requirements, thereby improving the reliability of the generation and arrangement of execution instructions.

[0031] S4 receives script generation instructions sent by the backend to the preset large language model based on the set of execution instructions, so that the preset large language model can determine the task type of each execution instruction, and generate the task program script corresponding to each execution instruction according to the task execution template corresponding to the task type of each execution instruction and the table structure information corresponding to the retrieved data table to be processed; it can be understood that: the third preset knowledge base stores the task types of several execution instructions, which are used to provide task classification basis for the large language model.

[0032] Specifically, the step of generating a task program script corresponding to each execution instruction based on the task execution template corresponding to the task type of each execution instruction and the table structure information corresponding to the retrieved data table to be processed includes the following steps: S401: For any given instruction, analyze the instruction and extract the table structure information corresponding to the instruction from the data table to be processed; this can be understood as: extracting the fields semantically related to the instruction from the data table to be processed, as well as the field values ​​corresponding to the fields.

[0033] S402, by using the access protocol corresponding to the execution instruction, obtain the task execution template corresponding to the task type of the execution instruction from the third preset knowledge base; wherein, the task execution template includes pre-built preset field insight rules, keyword checking rules and basic constraint rules.

[0034] S403, based on the structured requirements of the task execution template and the system prompt words that prohibit association set for the preset large language model, the corresponding program script is generated after semantic analysis of the task execution template through the preset large language model.

[0035] As described above, by using access protocols corresponding to different execution instructions, the corresponding task execution templates are obtained, preventing the program execution results from becoming chaotic when there are many tables and tasks. Furthermore, when the preset large language model generates task program scripts, it generates scripts based on the rules in the task execution templates. For example, in the preset field insight rules, ID cards need to have their saturation rate and whether the data type is numeric. In this process, the task execution template provides the preset large language model with the basis for script generation, utilizing the semantic understanding capabilities of the preset large language model to generate the final program script. The large language model is not allowed to make associations, ensuring that the final generated program script conforms to the system's data cleaning requirements and guarantees the quality of the cleaned data.

[0036] S5 sends each task script to the backend, allowing the backend to execute each script in an orderly manner, combining it with the actual data in the data table to be processed. This cleanses the data in the data table into standardized data and obtains the execution results of the processing tasks. Standardized data can be understood as data that conforms to local data format requirements after being cleaned according to the task script. For example, through field comparison, fields in the original data table to be processed are mapped to local fields, and then the data under the original fields is filled into the corresponding positions in the local table.

[0037] Furthermore, step S5 also includes the following steps: S501, when the amount of data in the data table to be processed is less than a preset data volume threshold, all data in the data table to be processed is cleaned. Those skilled in the art can set the preset data volume threshold according to actual needs, such as based on system memory limitations, which will not be elaborated here.

[0038] S502, when the amount of data in the data table to be processed is not less than a preset data amount threshold, a portion of the data in the data table to be processed is extracted according to a preset data extraction rule to generate several data subsets, which are then cleaned in batches; the data extraction rule is dynamically determined according to the user intent statement and the field business attributes of the data table to be processed.

[0039] As mentioned above, when the amount of data in the table to be processed is large, the execution of the script may easily lead to inaccurate data results. Therefore, it is necessary to clean the data in the table to be processed in batches. When cleaning in batches, the business attributes of the fields of the table to be processed and their combination with the user intent statement are taken into consideration, so that the batch division is more reasonable and meets the requirements of processing efficiency and the analytical value of each batch of data.

[0040] Furthermore, the preset data interception rules include the following steps: S5021, extract the keyword set from the user intent statement, and calculate the relevance weight of each field in the data table to be processed to the keyword set.

[0041] The relevance weight of any field to the keyword set meets the following conditions: W i =α×S i +β×N i +γ×T i Among them, W i S represents the relevance weight of the i-th field to the keyword set. i Let N be the semantic similarity between the i-th field and the keyword set. i T is the normalized value corresponding to the number of unique values ​​under the i-th field. i The preset importance score corresponds to the data type of the i-th field, where α, β, and γ are S... i N i and T i The corresponding preset weighting coefficients.

[0042] Preferably, α > 0.5.

[0043] Specifically, the number of unique values ​​under the i-th field refers to the number of field values ​​that do not contain duplicate values ​​among all field values ​​corresponding to the i-th field. During normalization, a common logarithmic function can be used to first reduce the numerical value of the count, and then the Softmax function can be used to normalize it to a range of 0-1.

[0044] As mentioned above, semantic similarity can characterize the relevance of a field to the user's intent. The more unique values ​​a field has, the more distinguishing information it usually contains, such as an ID number. Considering that date and numeric fields are usually more statistically significant than plain text fields in the analysis, different importance scores are set for different data types. By combining the above multiple dimensions, the selected fields are made more important and closely related to the user's intent.

[0045] S5022: Obtain the field business attributes of the field corresponding to the highest relevance weight. Based on the field business attributes and the corresponding sorted field values, sequentially extract data from the data table to be processed, generating several data subsets with a data volume less than a preset data volume threshold. For example, when the field values ​​under the field corresponding to the highest relevance weight are different months, after sorting the months in ascending or descending order according to statistical requirements, sequentially extract data from the data table to be processed. If the total data volume of the first three months is less than the preset data volume threshold, but the total data volume of the first four months is not less than the preset data volume threshold, then extract the data of the first three months of the month to form a data subset, and sequentially extract several data subsets.

[0046] S5023, according to the original order of several data subsets, sequentially combines the actual data of each data subset to run each task program script in an orderly manner.

[0047] As mentioned above, when calculating the relevance weights of fields and keyword sets, three dimensions were introduced: semantic similarity, the number of unique values ​​under the field, and the importance of data type. These were used as the basis for data batching. While ensuring the stable operation of the system, the system prioritized the extraction of a subset of data that was relevant to user intent and had high business representativeness for the first batch of processing. This enabled the system to efficiently control resources, ensure the analytical value of each batch of data, and provide an iteratively optimized cleaning benchmark for subsequent batches, thus achieving a balance between processing efficiency and result validity.

[0048] In a specific implementation, during step S5, when the backend runs each task program script in an orderly manner based on the actual data in the data table to be processed, the following steps are performed: S10: Run the task program script corresponding to the insight instruction, and perform insight analysis on the actual data of the data table to be processed according to the execution rules corresponding to the insight instruction to obtain the insight results; it can be understood as: after performing insight analysis according to various rules in the task execution template, the insight results are obtained.

[0049] S20, when the insight result indicates that the actual data of the data table to be processed meets the access requirements, the task program script corresponding to the cleaning instruction is run, the preset local standard data table rules are called, the fields of the data table to be processed are mapped and converted with the local standard fields, the data cleaning operation is performed, and standardized data conforming to the local standard format is generated.

[0050] Specifically, the local standard data table rules include field name mapping relationships, data type conversion rules, and data format normalization rules.

[0051] S30: After cleaning the data in the data table to be processed into standardized data, the fields of the standardized data are validated according to predefined field validation rules, and a validation report is generated based on the validation results; it can be understood that the validation report includes fields that have passed the validation and fields that have failed the validation, as well as the reasons for their failure.

[0052] Preferably, the field validation rules include independent validation rules based on field type and cross-validation rules based on relationships between fields. For example, a field may be an available field in itself, but a deregistration field in the local database.

[0053] As mentioned above, during data cleaning, data uploaded by different users are uniformly mapped and cleaned using local standard data table rules. Data from different data sources are stored locally with the same specifications, which unifies the data differences between various platforms and is beneficial to the quality and efficiency of subsequent data queries and data analysis.

[0054] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0055] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0056] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A data cleaning method based on a large language model, characterized in that, The method includes the following steps: S1 receives the content text and processing task corresponding to the data table to be processed input by the user, performs semantic parsing through a preset large language model, and generates the user's intent statement. S2, the constructed dataset to be processed is sent to the preset large language model, and the first preset knowledge base is invoked to guide the preset large language model to generate several unordered execution instructions corresponding to the processing task; the dataset to be processed includes the content text, field attribute information, processing task and user intent statement corresponding to the data table to be processed; the first preset knowledge base includes several preset instructions, execution rules and access protocols corresponding to each preset instruction; S3, call the preset instruction arrangement rules in the second preset knowledge base, so that the preset large language model can sort and classify several unordered execution instructions according to the preset instruction arrangement rules, generate an ordered set of execution instructions and send it to the backend; S4, receive the script generation instruction sent by the backend to the preset large language model based on the set of execution instructions, so that the preset large language model can determine the task type of each execution instruction, and generate the task program script corresponding to each execution instruction according to the task execution template corresponding to the task type of each execution instruction and the table structure information corresponding to the retrieved data table to be processed. S5 sends each task program script to the backend, so that the backend can run each task program script in an orderly manner in combination with the actual data in the data table to be processed, so as to clean the data in the data table to be processed into standardized data and obtain the execution results of the processing task.

2. The data cleaning method based on a large language model according to claim 1, characterized in that, Step S2 also includes the following steps: S201, When the dataset to be processed is received, check whether each key field corresponding to the data table to be processed is missing; the key field refers to the field that is semantically related to the processing task as determined by the preset large language model. S202, If any key fields are missing, provide the user with information about the missing fields. Conversely, calculate the data saturation corresponding to each key field; S203, when the data saturation corresponding to each key field is greater than the preset saturation threshold, the first preset knowledge base is called to guide the preset large language model and generate several unordered execution instructions corresponding to the processing task; otherwise, correction information is fed back to the user.

3. The data cleaning method based on a large language model according to claim 1, characterized in that, The instruction arrangement rules include grouping several execution instructions for the same data table into the same execution instruction set, and sorting several execution instructions within the execution instruction set according to the preset operation dependencies between the execution instructions.

4. The data cleaning method based on a large language model according to claim 1, characterized in that, In step S4, generating the task program script corresponding to each execution instruction based on the task execution template corresponding to the task type of each execution instruction and the table structure information corresponding to the retrieved data table to be processed includes the following steps: S401: For any given instruction, analyze the instruction and extract the table structure information corresponding to the instruction from the data table to be processed. S402, by using the access protocol corresponding to the execution instruction, obtain the task execution template corresponding to the task type of the execution instruction from the third preset knowledge base; wherein, the task execution template includes pre-built preset field insight rules, keyword checking rules and basic constraint rules; S403, based on the structured requirements of the task execution template and the system prompt words that prohibit association set for the preset large language model, the corresponding program script is generated after semantic analysis of the task execution template through the preset large language model.

5. The data cleaning method based on a large language model according to claim 1, characterized in that, Step S5 also includes the following steps: S501, when the amount of data in the data table to be processed is less than a preset data amount threshold, all data in the data table to be processed is cleaned. S502, when the amount of data in the data table to be processed is not less than the preset data amount threshold, a portion of the data in the data table to be processed is extracted according to the preset data extraction rules to generate several data subsets and then cleaned in batches. The data interception rules are dynamically determined based on the user's intent statement and the field business attributes of the data table to be processed.

6. The data cleaning method based on a large language model according to claim 5, characterized in that, The preset data extraction rules include the following steps: S5021, Extract the set of keywords from the user intent statement and calculate the relevance weight of each field in the data table to be processed to the set of keywords; The relevance weight of any field to the keyword set meets the following conditions: W i =α×S i +β×N i +γ×T i Among them, W i S represents the relevance weight of the i-th field to the keyword set. i Let N be the semantic similarity between the i-th field and the keyword set. i T is the normalized value corresponding to the number of unique values ​​under the i-th field. i The preset importance score corresponds to the data type of the i-th field, where α, β, and γ are S... i N i and T i The corresponding preset weighting coefficients; S5022: Obtain the field business attributes of the field corresponding to the highest relevance weight. Based on the field business attributes and the corresponding sorted field values, sequentially extract data from the data table to be processed to generate several data subsets with a data volume less than the preset data volume threshold. S5023, according to the original order of several data subsets, sequentially combines the actual data of each data subset to run each task program script in an orderly manner.

7. The data cleaning method based on a large language model according to claim 1, characterized in that, The aforementioned unordered preset instructions include insight instructions, cleaning instructions, and verification instructions; and in step S5, when the backend runs each task program script in an orderly manner in conjunction with the actual data in the data table to be processed, the following steps are implemented: S10, run the task program script corresponding to the insight instruction, and perform insight analysis on the actual data of the data table to be processed according to the execution rules corresponding to the insight instruction, and obtain the insight results; S20, when the insight result indicates that the actual data of the data table to be processed meets the access requirements, the task program script corresponding to the cleaning instruction is run, the preset local standard data table rules are called, the fields of the data table to be processed are mapped and converted with the local standard fields, the data cleaning operation is performed, and standardized data conforming to the local standard format is generated. The local standard data table rules include field name mapping relationships, data type conversion rules, and data format normalization rules; S30, after cleaning the data in the data table to be processed into standardized data, the fields of the standardized data are validated according to predefined field validation rules, and a validation report is generated based on the validation results; The field validation rules include independent validation rules based on field type and cross-validation rules based on the relationship between fields.

8. The data cleaning method based on a large language model according to claim 1, characterized in that, The preset large language model used in steps S1-S3 is the first preset intelligent agent in the preset large language model, and the preset large language model used in steps S4-S5 is the second preset intelligent agent in the preset large language model. The first and second intelligent agents are based on the same large language model instance, and their functions are differentiated by loading different system prompts and contextual knowledge.

9. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the data cleaning method based on a large language model as described in any one of claims 1-8.

10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.