A data cleaning method, device, product and medium based on large language model

Through the data cleaning method based on the large language model, the cleaning rules are automatically generated and adjusted, and the problem of difficult to guarantee the efficiency and quality of text data cleaning in the existing technology is solved, and efficient and automated cleaning of large-scale text data is achieved.

CN118520229BActive Publication Date: 2024-11-15BEIJING HAITIAN RUISHENG SCI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410991497.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-15
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

In the prior art, text data cleaning relies on manual fixed cleaning rules and cannot be adjusted adaptively, making it difficult to effectively eliminate noise items, especially in the case of increasing data scale and complexity, efficiency and quality are difficult to ensure.

Method used

The data cleaning method based on the large language model is adopted, and the data is fine-tuned by fine-tuning the data to be cleaned, matching cleaning rules are generated, and the data is cleaned. The cleaning rules are automatically adjusted using the evaluation results to realize adaptive data processing.

Benefits of technology

It realizes automatic cleaning of cleaning data, reduces manual participation, improves cleaning efficiency and quality, and can adaptively adjust cleaning rules according to different data expression forms, which is suitable for rapid cleaning of large-scale text data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118520229B_ABST
    Figure CN118520229B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a data cleaning method, device, product and medium based on a large language model, which relates to the field of data processing and is used to solve the problem of high degree of manual participation in the data cleaning process. The method includes: in response to the i-th cleaning of the data to be cleaned based on the large language model, fine-tuning the large language model based on the first rule data matching pair; generating a cleaning rule matching the current data to be cleaned based on the fine-tuned large language model; performing data cleaning on the current data to be cleaned to obtain the cleaned data, wherein the first rule data matching pair is determined based on the j-th cleaned data and the corresponding data to be cleaned and the j-th cleaning collection of cleaning rules, and the j-th cleaning collection of cleaning rules includes cleaning rules determined based on the j-th cleaned data and the corresponding data to be cleaned, i and j are positive integers, and i is greater than or equal to j. The method proposed in the present disclosure can improve the efficiency of data cleaning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a data cleaning method, device, product and medium based on a large language model. Background Art

[0002] In related technologies, the size and complexity of text data are growing continuously, and these data are usually obtained from a variety of data channels (e.g., social media, knowledge forums, news reports, scientific papers, etc.). However, it is understandable that since the data obtained from the above-mentioned multiple data channels usually contain noise, it is usually necessary to clean these data before using them to obtain higher quality data (e.g., data with lower text perplexity, etc.).

[0003] However, currently, data cleaning of the data to be cleaned is usually performed based on manually formulated cleaning rules. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data cleaning method, device, product and medium based on a large language model.

[0005] According to a first aspect of an embodiment of the present disclosure, in response to cleaning the data to be cleaned for the i-th time based on a large language model, the large language model used for the i-th cleaning data is fine-tuned based on a first rule data matching pair; wherein the first rule data matching pair is determined based on the j-th cleaned data, the data to be cleaned corresponding to the j-th cleaned data, and the cleaning rule collection corresponding to the j-th cleaning, the cleaning rule collection corresponding to the j-th cleaning includes the j-th new cleaning rule, the j-th new cleaning rule is a cleaning rule determined based on the j-th cleaned data and the data to be cleaned corresponding to the j-cleaned data, the i is a positive integer greater than or equal to 1, the j is a positive integer greater than or equal to 1, and the i is greater than or equal to the j; a cleaning rule matching the current data to be cleaned is generated based on the fine-tuned large language model, and data cleaning is performed on the current data to be cleaned to obtain first data; the first data is evaluated; and in response to the evaluation result of the first data meeting the requirements, the first data is determined to be the cleaned data.

[0006] In one embodiment, the large language model is determined in the following manner: based on a collection of cleaning rules, manually annotated cleaned data, and original data to be cleaned corresponding to the manually annotated cleaned data, a predicted large language model is trained to obtain an initial large language model; based on the initial large language model and the collection of cleaning rules, the original data to be cleaned is cleaned to obtain second data; based on the second data, the collection of cleaning rules, and the original data to be cleaned corresponding to the second data, a rule data matching pair is obtained, and based on the rule data matching pair, the initial large language model is fine-tuned to obtain a target large language model.

[0007] In one embodiment, the method further includes: generating new cleaning rules based on the data to be cleaned, the first data whose evaluation results meet the requirements, and the large language model rule generation model; adding the new cleaning rules to the cleaning rule collection to obtain a new rule collection; and updating the large language model based on the new rule collection.

[0008] In one embodiment, updating the large language model based on the new rule collection includes: generating a new rule-data matching pair based on the data to be cleaned, the new rule collection, and the first data whose evaluation results meet the requirements; and updating the large language model based on the new rule-data matching pair.

[0009] In one embodiment, the first data is evaluated based on at least one of the following: the degree of grammatical errors corresponding to the first data; the degree of spelling errors corresponding to the first data; the semantic accuracy corresponding to the first data; the completeness of the information corresponding to the first data; the text fluency corresponding to the first data; the vocabulary richness corresponding to the first data; and the accuracy of emotional color corresponding to the first data.

[0010] In one embodiment, the method further includes: before fine-tuning the large language model used for cleaning data for the i-th time based on the first rule data matching pair, the method further includes: determining that the usage time of the large language model reaches a usage time threshold; and / or determining that the amount of data cleaned by the large language model reaches a quantity threshold.

[0011] According to a second aspect of an embodiment of the present disclosure, a data cleaning device based on a large language model is provided, wherein a processing unit is used to fine-tune the large language model used for cleaning the data for the i-th time based on a first rule data matching pair when cleaning the data to be cleaned for the i-th time based on the large language model, wherein the first rule data matching pair is determined based on the j-th cleaned data, the data to be cleaned corresponding to the j-th cleaned data, and the cleaning rule collection corresponding to the j-th cleaning, the cleaning rule collection corresponding to the j-th cleaning includes the j-th new cleaning rule, and the j-th new cleaning rule is a cleaning rule determined based on the j-th cleaned data and the data to be cleaned corresponding to the j-cleaned data, wherein i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is greater than or equal to j; a data cleaning unit is used to generate a cleaning rule matching the current data to be cleaned based on the fine-tuned large language model, and perform data cleaning on the current data to be cleaned to obtain first data; and evaluate the first data, and when the evaluation result of the first data meets the requirement, determine that the first data is the cleaned data.

[0012] In one embodiment, the large language model is determined in the following manner: based on a collection of cleaning rules, manually annotated cleaned data, and original data to be cleaned corresponding to the manually annotated cleaned data, a predicted large language model is trained to obtain an initial large language model; based on the initial large language model and the collection of cleaning rules, the original data to be cleaned is cleaned to obtain second data; based on the second data, the collection of cleaning rules, and the original data to be cleaned corresponding to the second data, a rule data matching pair is obtained, and based on the rule data matching pair, the initial large language model is fine-tuned to obtain a target large language model.

[0013] In one embodiment, the data cleaning unit is further used to: generate new cleaning rules based on the data to be cleaned, the first data whose evaluation results meet the requirements, and the large language model rule generation model; add the new cleaning rules to the cleaning rule collection to obtain a new rule collection; and update the large language model based on the new rule collection.

[0014] In one embodiment, the data cleaning unit updates the large language model based on the new rule collection in the following manner: generating a new rule-data matching pair based on the data to be cleaned, the new rule collection, and the first data whose evaluation results meet the requirements; and updating the large language model based on the new rule-data matching pair.

[0015] In one embodiment, the processing unit evaluates the first data based on at least one of the following: the degree of grammatical errors corresponding to the first data; the degree of spelling errors corresponding to the first data; the semantic accuracy corresponding to the first data; the completeness of the information corresponding to the first data; the text fluency corresponding to the first data; the vocabulary richness corresponding to the first data; and the accuracy of emotional color corresponding to the first data.

[0016] In one embodiment, the large language model is also determined in the following manner: the processing unit is also used to: determine that the usage time of the large language model reaches a usage time threshold; and / or determine that the amount of cleaned data of the large language model reaches a quantity threshold.

[0017] According to a third aspect of an embodiment of the present disclosure, a data cleaning device based on a large language model is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the data cleaning method based on a large language model described in the first aspect or any one of the embodiments of the first aspect.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the data cleaning method based on a large language model described in the first aspect or any one of the embodiments of the first aspect.

[0019] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the data cleaning method based on a large language model described in the first aspect or any one of the embodiments of the first aspect is implemented.

[0020] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: In the data cleaning method based on the large language model proposed in the embodiment of the present disclosure, by responding to the large language model before performing the i-th data cleaning, the large language model is fine-tuned through the first rule data matching pair. Since the first rule data matching pair is determined by the j-th cleaned data, the data to be cleaned corresponding to the j-th cleaned data, and the collection of cleaning rules corresponding to the j-th cleaning, the large language model can generate a cleaning rule matching the current data to be cleaned according to the current data to be cleaned, perform data cleaning, obtain the first data after cleaning, and then perform a corresponding evaluation on the first data, such as an evaluation of the cleaning quality. And the first data whose evaluation result meets the requirements is used as the cleaned data. On the one hand, the automatic cleaning of the data to be cleaned is realized through the large language model, which reduces the degree of manual participation, thereby realizing the automation of data cleaning. On the other hand, it can be determined whether the current cleaning effect meets the requirements based on the evaluation of the cleaned data, so that the cleaned data that meets the requirements can be determined.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] Figure 1 The present invention is a flowchart of a data cleaning method based on a large language model according to an exemplary embodiment.

[0024] Figure 2 The figure is a flowchart of a large language model training method according to an exemplary embodiment.

[0025] Figure 3A The figure is a flowchart of a method for updating a large language model according to an exemplary embodiment.

[0026] Figure 3B The figure is a flowchart of a method for updating a large language model according to an exemplary embodiment.

[0027] Figure 4 The present invention is a schematic diagram of an operation flow of a data cleaning system based on a large language model according to an exemplary embodiment.

[0028] Figure 5A The present invention is a flowchart of a preprocessing method of a data cleaning system based on a large language model according to an exemplary embodiment.

[0029] Figure 5BThe figure is a flow chart of a method for updating a large language model cleaning engine according to an exemplary embodiment.

[0030] Figure 5C The invention is a flowchart showing a method for data cleaning application based on a large language model according to an exemplary embodiment.

[0031] Figure 5D The figure is a flowchart of a method for updating a rule collection according to an exemplary embodiment.

[0032] Figure 6 The present invention is a block diagram of a data cleaning device based on a large language model according to an exemplary embodiment.

[0033] Figure 7 The present invention is a block diagram of a data cleaning device based on a large language model according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0035] In related technologies, the size and complexity of text data are growing continuously, and these data are usually obtained from a variety of data channels (e.g., social media, knowledge forums, news reports, scientific papers, etc.). However, it is understandable that since the data obtained from the above-mentioned multiple data channels usually contain noise, it is usually necessary to clean these data before using them to obtain higher quality data (e.g., data with lower text perplexity, etc.).

[0036] However, currently, data cleaning of the data to be cleaned is usually performed based on manually formulated cleaning rules, which are usually relatively fixed and cannot be adaptively adjusted according to different data representation forms.

[0037] For example, taking the text type of data to be cleaned as an example, the manually formulated cleaning rule is "retain sentences with a length greater than 10 characters", and the first text data to be cleaned obtained is, for example, "We will continue to follow up on this incident. Friendly link / nAA News / n / r / nHow to solve the problem? Contact number xxx". Among them, "We will continue to follow up on this incident" will be considered by users as valid data, and "Friendly link / nAA News / n / r / nHow to solve the problem? Contact number xxx" will be considered by users as a noise item. Then, the cleaned data obtained by the cleaning rule should be "We will continue to follow up on this incident", which means that the noise items are eliminated. If the second text data to be cleaned obtained is, for example, "We will continue to follow up on this incident. How to solve the problem? Contact number XXXYYZZZ". Among them, "We will continue to follow up on this incident" will be considered by users as valid data, and "How to solve the problem? Contact number XXXYYZZZ" will be considered by users as a noise item. If the cleaning rules remain unchanged, the second text data to be cleaned may be cleaned as "How to solve the problem? Contact number XXXYYZZZ", which means that the noise item is not removed. In this case, if you want to remove the noise item, you need to adaptively change the cleaning rules for this text to be cleaned (for example, change it to "remove meaningless sending links or advertisements", etc.).

[0038] Based on this, in order to solve the above technical problems, the disclosed embodiment proposes a data cleaning method based on a large language model (LLM). By performing data cleaning on the data to be cleaned based on the large language model, the cleaned data is obtained. The large language model has the ability to generate cleaning rules that match the data to be cleaned, and the ability to clean the data; the cleaned data is evaluated; in response to the evaluation result of the cleaned data meeting the requirements, the cleaned data is determined to be the final data. By using a large language model to clean the data, and the large language model has the ability to generate cleaning rules that match the data to be cleaned, the degree of manual participation is reduced, and the automation of data cleaning is realized. In addition, when faced with a large order of magnitude of data to be cleaned, the cleaning work can be completed more quickly than manual cleaning.

[0039] For ease of understanding, the data to be cleaned in the following embodiments are illustrated by taking text-type data to be cleaned as an example.

[0040] Figure 1 is a flow chart of a data cleaning method based on a large language model according to an exemplary embodiment. Figure 1 As shown, the data cleaning method includes the following steps.

[0041] In step S11 , in response to the i-th cleaning of the data to be cleaned based on the large language model, the large language model used for the i-th cleaning of the data is fine-tuned based on the first rule data matching pair.

[0042] In step S12, a cleaning rule matching the current data to be cleaned is generated based on the fine-tuned large language model, and data cleaning is performed on the current data to be cleaned to obtain first data.

[0043] In step S13 , the first data is evaluated.

[0044] In step S14 , in response to the evaluation result of the first data satisfying the requirement, it is determined that the first data is cleaned data.

[0045] Among them, the first rule data matching pair is determined based on the j-th cleaned data, the data to be cleaned corresponding to the j-th cleaned data, and the cleaning rule collection corresponding to the j-th cleaning, the cleaning rule collection corresponding to the j-th cleaning includes the j-th new cleaning rule, the j-th new cleaning rule is a cleaning rule determined based on the j-th cleaned data and the data to be cleaned corresponding to the j-th cleaned data, i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is greater than or equal to j.

[0046] In the disclosed embodiment, when the large language model cleans the data to be cleaned for the i-th time, the large language model is fine-tuned based on the first rule data matching, so that the fine-tuned large language model has the ability to generate cleaning rules that match the data to be cleaned, and the i-th data is cleaned based on the fine-tuned large model to obtain the first data after cleaning, and then the first data is evaluated accordingly, such as the evaluation of the cleaning quality. And the first data whose evaluation result meets the requirements is used as the cleaned data. On the one hand, the automatic cleaning of the data to be cleaned is realized through the large language model, and the corresponding cleaning rules can be generated by using the first cleaned data and the corresponding data to be cleaned for fine-tuning, so that the large language model can be adaptively updated, reducing the degree of manual participation, thereby realizing the automation of data cleaning. On the other hand, it can be determined whether the current cleaning effect meets the requirements based on the evaluation of the cleaned data, so that the cleaned data that meets the requirements can be determined.

[0047] For ease of understanding, in the embodiment of the present disclosure, the cleaning rule generated by the large language model to match the data to be cleaned is referred to as the first cleaning rule.

[0048] It is understandable that the data to be cleaned is cleaned by the cleaning rules to obtain the cleaned data. The cleaning rules may include at least one of the following: removing duplicate data, filling missing values, correcting erroneous values, standardizing data formats, converting data types, filtering irrelevant data, and identifying and processing abnormal values.

[0049] Exemplarily, if the data to be cleaned is “The weather is nice, go fly a kite, go fly a kite, go fly a kite.” Data cleaning is performed based on deduplicated data, and the cleaned data may be “The weather is nice, go fly a kite.”

[0050] As another example, if the data to be cleaned is "Listen to opinions, continue to improve, and give a good review to a certain anti-virus software again! \nJager·A month B day·C year a certain anti-virus software·Cache duration set D times read\L><Website navigation》Friendly links》xxxxxxxxx", it should be noted that. In this example, the data to be cleaned, "nJager·A month B day·C year a certain anti-virus software·Cache duration set D times read\L><Website navigation》Friendly links》xxxxxxxxx" is irrelevant data. Then, the data to be cleaned is cleaned based on "filtering irrelevant data", and the cleaned data obtained can be "Listen to opinions, continue to improve, and give a good review to a certain anti-virus software again!". It can be seen that there are multiple sections of content with weak semantic correlation in the data to be cleaned (for example, "A month B day" has weak semantic correlation with the previous and next content), which will make the text of the data to be cleaned look more confused as a whole.

[0051] It should be noted that text perplexity can be understood as the degree of difficulty a person has in understanding an article in a reading comprehension task. The lower the text perplexity, the stronger the model's ability to understand and the easier it is to process the text; the higher the text perplexity, the weaker the model's ability to understand and the more difficult it is to process the text. Based on this, we can use text perplexity to determine whether the text needs to be screened out. When the perplexity of the text exceeds the set threshold, it indicates that the text is too complex for the model and should be screened out. As for the degree of fit between the cleaned data and the cleaning rules, it can be regarded as the degree of match or fit between the rules applied in the cleaning process and the cleaned data. The higher the degree of fit, it can be considered that the cleaning rules can more effectively identify and handle problems in the data.

[0052] However, it is understandable that since the cleaned data is suitable for other text processing operations or model training operations, the evaluation results of the cleaned data need to have quality requirements. In order to identify the cleaned data that meets the quality requirements, the corresponding representation can be set to determine whether the quality of the cleaned data meets the quality requirements.

[0053] In some embodiments, for example, the evaluation result after cleaning meets requirements (such as quality requirements), including: the text perplexity of the cleaned data is less than or equal to the text perplexity threshold, and the fitness between the cleaned data and the first cleaning rule is greater than or equal to the fitness threshold.

[0054] In some embodiments, the terms "evaluation results", "evaluation", "quality", "quality results" and the like can be used interchangeably.

[0055] Based on this, we can determine whether the text perplexity of the cleaned data meets the perplexity requirements by setting a perplexity threshold. If the text perplexity of the cleaned data is less than or equal to the perplexity threshold, it can be considered that the text perplexity of the cleaned data meets the requirements.

[0056] Based on this, through the description of meeting the requirements in the above embodiments, it can be determined that if the evaluation result of the first data meets the requirements, it can be considered that the first cleaning rule generated by the rule model has high adaptability to the first data. It can also be determined to a certain extent that the first cleaning rule generated by the large language model is a cleaning rule that meets the data cleaning requirements.

[0057] However, in the actual cleaning process, the first data evaluation result obtained after cleaning may not necessarily meet the requirements (for example, in the initial application stage after the large language model is trained). In this case, it can be processed by the following implementation method.

[0058] In some embodiments, in response to the evaluation result of the first data not meeting the quality requirement, the first data is discarded, wherein the quality meeting the requirement includes, for example: the score of the first data is greater than or equal to a score threshold.

[0059] The score for the first data is an expression of the evaluation result obtained after evaluating the first data. For example, a high score (or greater than or equal to the corresponding score threshold) indicates that the evaluation result meets the requirements, and a low score (or the score threshold corresponding to rain) indicates that the evaluation result does not meet the requirements.

[0060] The evaluation of the first data can be achieved based on at least one of the following: the degree of grammatical errors corresponding to the first data, the degree of spelling errors corresponding to the first data, the semantic accuracy corresponding to the first data, the completeness of the information corresponding to the first data, the text fluency corresponding to the first data, the vocabulary richness corresponding to the first data, and the accuracy of the emotional color corresponding to the first data.

[0061] Among them, the degree of grammatical errors corresponding to the cleaned data can be, for example, whether there are grammatical errors, or the proportion of grammatical errors. The degree of spelling errors corresponding to the cleaned data can be, for example, whether there are spelling errors, or the proportion of spelling errors. The semantic accuracy corresponding to the cleaned data can be, for example, whether the semantics of the cleaned data is clearly expressed, logically coherent, and in line with the context. The text fluency corresponding to the cleaned data can be, for example, whether the cleaned data is fluent and easy to read (for example, the higher the fluency, the more fluent and easy to read), and whether appropriate language expressions are used. The vocabulary richness (or vocabulary richness) corresponding to the cleaned data can be, for example, the diversity and richness of the cleaned data, which can be measured by calculating the number or type of different words in the cleaned data. The emotional color accuracy corresponding to the cleaned data can be, for example, the accuracy between the emotional color (for example, commendatory color, or derogatory color, etc.) contained in the cleaned data and the expected emotional color.

[0062] It is understandable that, during the data cleaning process, if the data quality obtained after the data to be cleaned is cleaned based on the first cleaning rule does not meet the quality requirements, the data will be discarded. If the data quality requirements are met, it will be used normally.

[0063] In some embodiments, the quality does not meet the requirements, for example, including: the text perplexity of the cleaned data is greater than the text perplexity threshold.

[0064] However, in some application scenarios, if the evaluation result of the first data does not meet the quality requirements, it can be inferred that it may be because the first cleaning rule generated is not applicable to the data cleaning of the current data to be cleaned. Therefore, in this scenario, the data can be cleaned again by generating a new cleaning rule so that the quality of the cleaned data meets the quality requirements.

[0065] In some scenarios, the data cleaning process can use the first cleaning rule to clean the data, or the first cleaning rule can be used as an auxiliary cleaning rule to clean the data.

[0066] In some embodiments, the large language model may be trained in the following manner. Figure 2 is a flowchart of a large language model training method according to an exemplary embodiment. Figure 2 As shown, the large language model training method includes the following steps.

[0067] In step S21, based on the collection of cleaning rules, the manually annotated cleaned data, and the original data to be cleaned corresponding to the manually annotated cleaned data, the prediction large language model is trained to obtain an initial large language model.

[0068] In step S22, the original data to be cleaned is cleaned based on the initial large language model and the collection of cleaning rules to obtain second data.

[0069] In step S23, based on the second data, the collection of cleaning rules and the original data to be cleaned corresponding to the second data, a rule data matching pair is obtained, and based on the rule data matching pair, the initial large language model is fine-tuned to obtain a target large language model.

[0070] It should be noted that the predicted large language model can be understood as a large language model that has not been trained.

[0071] In the disclosed embodiment, by initially training the predicted large language model based on the cleaning rule collection, manually annotated cleaned data, and the original data to be cleaned to obtain the initial large language model, the large language model can be enabled to have the ability to manually clean the data to be cleaned based on the cleaning rules, and then the cleaning of the data to be cleaned can be realized based on the large language model. And by using the initial large language model to clean the data to obtain the second data combined with the cleaning rule collection and the original data to perform rule data matching, a rule data matching pair is obtained, and then the initial large language model is fine-tuned based on the matching pair and repeated multiple times, thereby obtaining the final target large language model. For the large language model, both its initialization training and fine-tuning update stages can reduce the degree of human participation and realize the automation of the large language model determination process.

[0072] In some embodiments, before fine-tuning the large language model used for cleaning data for the i-th time based on the first rule data matching pair, the method also includes: determining that the usage time of the large language model reaches a usage time threshold; and / or determining that the amount of data cleaned by the large language model reaches a quantity threshold.

[0073] In the disclosed embodiment, by determining the triggering conditions for fine-tuning the large language model, the large language model can adaptively adjust parameters during use, thereby making the output results of the large language model more accurate during application.

[0074] In some embodiments, a cleaning rule collection can be understood as a collection of data cleaning rules, wherein the stored cleaning rules can be used for training a large language model and data cleaning based on a large language model. For example, the initial cleaning rules stored in the cleaning rule collection are cleaning rules for obtaining an initial large language model by training a large language model, and the initial cleaning rules stored in the cleaning rule collection may include, for example, an initial rule template for initial training of a large language model. In the process of data cleaning by a large language model, the first cleaning rule for data cleaning can also be determined through the rule collection (for example, selecting a suitable initial cleaning rule as the first cleaning rule, or generating a first cleaning rule through the initial cleaning rule, etc.). It can be understood that cleaning the data to be cleaned based on the rule collection can be understood as using the cleaning rules in the rule collection (for example, the initial cleaning rules) to clean the data to be cleaned.

[0075] In some embodiments, the rule-data matching pair may consist of the original data to be cleaned, the second data corresponding to the original data to be cleaned, and the cleaning rule.

[0076] Exemplarily, the original data to be cleaned may be "Listen to opinions, continue to improve, and give a good review to a certain antivirus software again! \nJager·A month B day·C year a certain antivirus software·cache duration set D times read\L><Website navigation》Friendly links》xxxxxxxxx". The "cleaning rules" are: "Retain sentences with a length greater than 10 characters" and "Delete meaningless paragraphs containing date, time, user name and other original data". The cleaned data obtained is "Listen to opinions, continue to improve, and give a good review to a certain antivirus software again!", then the above three can constitute a rule data matching pair.

[0077] Furthermore, in some embodiments, the rule data matching pair may further include a score for the second data. Continuing with the above embodiment, for example, the final score may be, for example, text quality: 9, rule 1 fitness: 10, rule 2 fitness: 10, and self-generated rule validity: "valid".

[0078] It should be noted that the scoring mechanism can be used to determine whether the result of data cleaning performed by the large language model is a cleaning result that meets the quality requirements. For example, the scoring value can be set to different scoring levels of 1 to 10, and it is stipulated that when the scoring value is less than a certain scoring threshold (for example, less than 5 points), it can be determined that the quality evaluation is a manually annotated cleaning result that does not meet the quality requirements, and when the scoring value is greater than or equal to a certain scoring threshold (for example, greater than or equal to 5 points), it can be determined that the quality evaluation is a manually annotated cleaning result that meets the quality requirements. It can be understood that the determination of the scoring value can refer to the relevant description for determining the tie in the above embodiment, and will not be repeated here one by one.

[0079] In some embodiments, the initial large language model is fine-tuned, for example, based on a scoring mechanism.

[0080] In some embodiments, data cleaning is performed based on a large language model, and / or during the large language model determination process, the large language model generates new cleaning rules, wherein the generated new cleaning rules are used for large language model data cleaning and / or iterative updating of the large language model.

[0081] It is understandable that during the data cleaning and / or determination process of the large language model, the large language model will continuously output the data cleaning results. And the new cleaning rules can be newly generated based on the data cleaning results and the corresponding data to be cleaned. For example, different from the cleaning rules in the cleaning rule collection used in the data cleaning process, and / or different from the cleaning rules in the cleaning rule collection used in the process of determining the large language model.

[0082] For example, the implementation method can be as follows: Figure 3A shown. Figure 3A The invention is a flowchart of a method for updating a large language model according to an exemplary embodiment. The method includes the following steps.

[0083] In step S31, new cleaning rules are generated based on the data to be cleaned, the first data whose evaluation results meet the requirements, and the large language model rule generation model.

[0084] In step S32, the new cleaning rule is added to the cleaning rule collection to obtain a new rule collection.

[0085] In step S33, the large language model is updated based on the new rule collection.

[0086] In the disclosed embodiment, by generating new cleaning rules based on the data to be cleaned, the first data whose evaluation results meet the requirements, and the large language model rule generation model, a correspondence between the target cleaning rule and the cleaned data that meets the quality requirements can be established. And the large language model is trained based on the target cleaning rule, so that the trained large language model can perform better data cleaning during the data cleaning process, making the probability that the output cleaned data meets the quality requirements higher.

[0087] It should be noted that the "new rule collection" refers to a collection with new cleaning rules added, which can be the state after the new cleaning rule collection is added to the original rule collection, or it can be a rule collection different from the original rule collection. For example, in the above embodiment, the j-th new cleaning rule is added to the cleaning rule collection corresponding to the j-th cleaning, and the resulting rule collection can be understood as the "new rule collection".

[0088] For example, the generation of new rules can be: if the data to be cleaned is "The weather is good, go fly a kite, go fly a kite", and the first data is "The weather is good, go fly a kite", then the target cleaning rule can be inferred from the two, for example, the inferred target cleaning rule is "remove duplicate content".

[0089] In some embodiments, the newly generated new cleaning rule may be semantically identical to the first cleaning rule or the second cleaning rule and have the same text form. Therefore, in order to avoid wasting storage space, the cleaning rules in the rule collection can be regularly determined, and the newly generated rules that are semantically identical and have the same text form can be deduplicated, while the new cleaning rules that are semantically identical but have different text forms can be retained. In this way, cleaning rules with the same semantics but different representation forms (such as word order) can be added to the cleaning rule set, increasing the diversity of rule types and thereby improving the performance of the large language model.

[0090] Update the large language model according to the new set of rules, for example Figure 3B shown. Figure 3B The invention is a flowchart of a method for updating a large language model according to an exemplary embodiment. The method includes the following steps.

[0091] In step S41, a new rule-data matching pair is generated based on the data to be cleaned, the new rule set, and the first data whose evaluation results meet the requirements.

[0092] In step S42, the large language model is updated based on the new rule data matching pair.

[0093] In the disclosed embodiment, since the first data is the data cleaned by the large language model, a new rule collection is determined by the first data and the data to be cleaned that meet the conditions based on the evaluation results. A new rule-data matching pair is generated based on the new rule collection, and the large language model is updated based on the new rule-data matching pair. The large language model can be self-iterated based on the data in the reasoning application process, reducing the process of manual participation in the update. To a certain extent, it also realizes the optimization and update of the cleaning rules in the rule collection, which is convenient for the subsequent improvement of the model cleaning results.

[0094] In some embodiments, for iterative updates of large language models, update conditions can be customized according to needs, for example, when the number of data cleaning times of the model reaches a threshold number, the large language model is iteratively updated. Or when the score of the data cleaning result does not meet the expected requirements (for example, when the number of low evaluation results reaches a threshold number), the large language model is iteratively updated.

[0095] Through the description of the above embodiments, the degree of manual participation in the initial training stage, data cleaning stage and iterative update stage of the large language model is reduced, thereby alleviating the manual workload.

[0096] Based on the same concept, the embodiment of the present disclosure also proposes a data cleaning system. The data cleaning system has adaptive learning capabilities and can dynamically adjust the cleaning strategy according to the characteristics of the data to achieve a better cleaning effect.

[0097] For ease of understanding, the present disclosure will be based on Figure 4 A description of the data cleaning system is given. Figure 4 FIG. 1 is a schematic diagram of an operation flow of a data cleaning system based on a large language model according to an exemplary embodiment. Figure 4 As shown in the figure, the data cleaning system can be divided into the following four parts according to the arrow description:

[0098] The part with manual participation (the route composed of filled arrows and dotted lines), the large language model cleaning engine update part (the route composed of single-line arrows and solid lines), the data cleaning application part (the route composed of single-line arrows and dotted lines) and the rule collection update part (the route composed of filled arrows and solid lines).

[0099] exist Figure 4 In the process, the manual participation part obtains manual cleansing data by manually processing the original data based on the rule collection in the rule collection, and constructs training samples for the large language model cleaning engine based on the original data, the rule collection and the manual cleansing data, trains the initial large language model cleaning engine, and scores the results of the original data cleansing according to the initial large language model cleaning engine, iterates based on the scoring, and obtains the first version of the large language model cleaning engine. The first version of the large language model cleaning engine is put into use, the original data is cleaned, the cleaned data obtained is scored, and the quality is judged based on the scoring results. The cleaned data that does not meet the quality requirements is discarded, and the data that meets the quality requirements (i.e., high-quality data) is determined as the final data. After obtaining the final data, new rules can also be generated through the large language model rule engine using the original data corresponding to the final data, and the generated new rules are put into the rule collection for subsequent iteration or cleaning of the large language model cleaning engine.

[0100] The following will explain each part of the route in the above data cleaning system from (1) to (4).

[0101] (1) The manual part can be understood as the preprocessing process of the data cleaning system (e.g. Figure 4In the manual processing based on the original data and the rule set to obtain the manually cleaned data), see Figure 5A Related description, Figure 5A is a flowchart of a preprocessing method of a data cleaning system based on a large language model according to an exemplary embodiment. Figure 5A As shown, the following steps may be included:

[0102] In step S51 - 1 , a rule collection is set.

[0103] In step S51 - 2 , the original data is cleaned to obtain cleaned manually cleaned data.

[0104] In some embodiments, manual data cleaning of raw data may be understood as manual cleaning of raw data based on initial rules in a rule collection.

[0105] In some embodiments, the execution order of steps S51 - 1 and S51 - 2 can be interchangeable.

[0106] It should be noted that the original data, which can be understood as the data to be cleaned, contains, for example, meaningless sentences, symbols, and Hyper Text Markup Language (HTML). For the relevant description of the data to be cleaned, please refer to the above Figures 1 to 3B The description of the related embodiments will not be repeated here one by one.

[0107] The rule collection can be understood as a collection of general or specific rules used to clean data. The collection includes the initial version preset by humans and the rules updated by the subsequent large language model cleaning engine. Figures 1 to 3B The description of the related embodiments will not be repeated here one by one. It should be noted that Figure 4 The “large language model cleaning engine” in other embodiments may also be referred to as the “LLM cleaning engine”.

[0108] In the embodiments of the present disclosure, the manually cleaned data obtained by the manual participation part can be used as the first label to perform corresponding training on the large language model cleaning engine.

[0109] (2) Cleaning the engine update for large language models (e.g. Figure 4 The original data, rule collection, manually cleaned data, and cleaned data scores are used as training samples to update the large language model cleaning engine), see Figure 5B Related description, Figure 5B is a flow chart of a large language model cleaning engine update method according to an exemplary embodiment. Figure 5B As shown, the method comprises the following steps:

[0110] In step S52-1, training samples are determined based on one or more of the original data, the rule collection, the manually cleaned data, and the cleaned data.

[0111] In step S52 - 2 , the large language model cleaning engine is updated based on the determined training samples.

[0112] It should be noted that the update of the large language model cleaning engine includes initialization training and iterative update processes. For example, the initialization training of the large language model cleaning engine is performed by determining the training samples used for initialization training through one or more of the original data, rule collection, manually cleaned data, and cleaned data, or the iterative update of the large language model cleaning engine is performed by determining the training samples used for iterative update through one or more of the original data, rule collection, manually cleaned data, and cleaned data. For the construction of each sample, please refer to Figure 3A as well as Figure 3B The relevant embodiments are described in detail herein and will not be described in detail here.

[0113] For example, the "original data" is: "Listen to opinions, continue to improve, and give a good review to a certain anti-virus software again! \nJager·A month B day·C year a certain anti-virus software·cache duration setting D times read\L><Website navigation》Friendly links》xxxxxxxxx". The "cleaning rules" are: "Retain sentences with a length greater than 10 characters" and "Delete meaningless paragraphs containing date, time, user name and other original data". The cleaned data obtained is "Listen to opinions, continue to improve, and give a good review to a certain anti-virus software again!".

[0114] The final rating may be, for example, text quality: 9, rule 1 fitness: 10, rule 2 fitness: 10, and self-generated rule validity: "valid".

[0115] It should be noted that for the validity of self-generated rules, for example, when the text quality and the adaptability of each rule meet a certain threshold, "valid" can be generated accordingly. Text quality can be understood as the perplexity of the text, and the higher the text quality, the lower the perplexity of the text.

[0116] In some embodiments, for substantially the same data cleaning rules, there may be differences in their textual expressions. Therefore, in the process of constructing training samples, after obtaining the elements used to construct the training samples (one or more of the original data, the rule set, the manually cleaned data, and the cleaned data), multiple different prompt templates may need to be set (the outputs of each prompt template in different prompt templates may be the same).

[0117] Continuing with the above embodiment, when the "original data" is the same, if the "cleaning rules" generated by the prompt template are: "If the number of words in a sentence does not exceed 10, the sentence should be discarded" and "Remove all meaningless paragraphs in the text, such as date, author name, number of readings and other information". It can be understood that although the cleaning rules generated by the prompt in this embodiment are different in form from the cleaning rules in the above embodiment, the operations performed by the two are essentially the same, so the cleaned text obtained by data cleaning based on the current "cleaning rules" should be "Listen to opinions, continue to improve, and give a good review to a certain antivirus software again!" The same as the cleaned text in the above embodiment. The corresponding scores can be, for example, text quality: 9, rule 1 fitness: 10, rule 2 fitness: 10, self-generated rule validity: "valid".

[0118] In some embodiments, a plurality of prompt templates with the same output in different forms may be formed into a semantic fragment tree (SFT) set for constructing a final large language model.

[0119] In some embodiments, since the prompt can be set to multiple versions, the data for fine-tuning can be input to the large language model cleaning engine through the prompt generator. The data sent to the large language model cleaning engine for fine-tuning can be high-quality text or low-quality text. The evaluation of quality can be determined, for example, by scoring. As for the scoring method, please refer to the relevant description in the above embodiment, which will not be repeated here.

[0120] (3) For data cleaning applications (e.g. Figure 4 In the process of inputting the original data into the large language model cleaning engine for cleaning, after obtaining the cleaned data, the cleaned data is scored and the quality of the cleaned data is judged. After the quality judgment, the cleaned data that does not meet the quality requirements is discarded, and the cleaned data that meets the quality requirements is determined to be high-quality data and output as the final data). Figure 5C Related description, Figure 5Cis a flow chart of a method for data cleaning application based on a large language model according to an exemplary embodiment. Figure 5C As shown, the following steps may be included:

[0121] In step S53 - 1 , the large language model cleaning engine is called to perform data cleaning on the original data.

[0122] In step S53-2, the cleaned data is scored.

[0123] In step S53 - 3 , quality judgment is performed based on the cleaned data scores, high-quality data is used as final data, and low-quality data is discarded.

[0124] In some embodiments, the cleaned data is scored, for example, it can be scored based on whether the items meet the quality requirements, and high-quality data can be considered as data that meets the quality requirements, and low-quality data can be considered as data that does not meet the quality requirements.

[0125] It should be noted that for the quality requirements, please refer to the above embodiment. Figures 1 to 3B The description of the related embodiments will not be repeated here one by one.

[0126] (4) Update the rule collection (e.g. Figure 4 The raw data and high-quality data in the big language model rule engine are sent to the big language model rule engine, and new rules are generated by the big language model rule engine, and then the generated new rules are added to the rule collection), where the "big language rule engine" can also be called the "LLM rule engine". Figure 5D Related description, Figure 5D is a flow chart of a method for updating a rule collection according to an exemplary embodiment. Figure 5D As shown, the following steps may be included:

[0127] In step S54 - 1 , the original data and the high-quality data are input into the large language model rule engine.

[0128] In step S54 - 2 , the large language model rule engine is called to output new rules and update them into the rule collection.

[0129] In some embodiments, when a large language model rule engine outputs a new rule, the new rule may be generated, for example, by a few-shot-based prediction approach.

[0130] Exemplarily, for determining the corresponding new rules based on the original data and the cleaned data, the process may be, for example: evaluating the feasibility of the current cleaning rules, and based on the analysis results, proposing optimization or modification suggestions, and then generating a more effective rule list. For example, the "original data" is: "Listen to opinions, continue to improve, and give a positive review to a certain antivirus software again! \nJager·A month B day·C year a certain antivirus software·cache duration set D times read\L><Website navigation》Friendly links》xxxxxxxxx". The cleaned data is "Listen to opinions, continue to improve, and give a positive review to a certain antivirus software again!". The rules in the current rule collection are, for example, "If the number of words in a sentence does not exceed 10, the sentence should be discarded" and "Remove all meaningless paragraphs in the text, such as date, author name, number of readings, etc." Call the large language model rule engine to determine the following matters: confirm whether the existing rules are applicable to the original data and explain the reasons; if the existing rules are not feasible or need to be improved, please propose modification suggestions or new rules; ensure that the final rule list can effectively clean the text while retaining valuable information. By calling the rules of the large language model rule engine to output the response, it is possible to determine whether new data cleaning rules need to be updated based on the relationship between the original data and the cleaned data. If there are data cleaning rules that need to be updated, the new data cleaning rules will be updated to the rule collection.

[0131] The data cleaning system proposed in the above embodiment has adaptive learning capability and can dynamically adjust the cleaning strategy according to the characteristics of the data to achieve better cleaning effect.

[0132] Based on the same concept, the embodiment of the present disclosure also provides a data cleaning device based on a large language model.

[0133] It is understandable that in order to achieve the above functions, the data cleaning device based on the large language model provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0134] Figure 6 FIG. 1 is a block diagram of a data cleaning device based on a large language model according to an exemplary embodiment. Figure 6 The device 100 includes a processing unit 101 and a data cleaning unit 102 .

[0135] The processing unit 101 is used to fine-tune the large language model used for the i-th cleaning data based on the first rule data matching pair when the data to be cleaned is cleaned for the i-th time based on the large language model, wherein the first rule data matching pair is determined based on the j-th cleaned data, the data to be cleaned corresponding to the j-th cleaned data, and the cleaning rule collection corresponding to the j-th cleaning, the cleaning rule collection corresponding to the j-th cleaning includes the j-th new cleaning rule, the j-th new cleaning rule is a cleaning rule determined based on the j-th cleaned data and the data to be cleaned corresponding to the j-th cleaned data, i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is greater than or equal to j.

[0136] The data cleaning unit 102 is used to generate cleaning rules that match the current data to be cleaned based on the fine-tuned large language model, and perform data cleaning on the current data to be cleaned to obtain first data; and evaluate the first data, and when the evaluation result of the first data meets the requirements, determine that the first data is cleaned data.

[0137] In some embodiments, the large language model is determined in the following manner: based on the cleaning rule collection, manually annotated cleaned data, and original data to be cleaned, the predicted large language model is trained to obtain an initial large language model; based on the initial large language model and the cleaning rule collection, the original data to be cleaned is cleaned to obtain second data; based on the second data, the cleaning rule collection, and the original data to be cleaned, a rule data matching pair is obtained, and based on the rule data matching pair, the initial large language model is fine-tuned to obtain a fine-tuned large language model; repeatedly performing the process of cleaning the original data to be cleaned based on the fine-tuned large language model and the cleaning rule collection, and generating rule data matching pairs based on the data cleaned by the optimized large language model, the cleaning rule collection, and the original data to be cleaned, and fine-tuning the fine-tuned large language model again until the large language model meets the target requirements to obtain a target large language model.

[0138] In some embodiments, the processing unit 101 is further configured to update the large language model based on the data to be cleaned and the first data.

[0139] In some embodiments, the data cleaning unit 102 is further used to: generate new cleaning rules based on the data to be cleaned, the first data whose evaluation results meet the requirements, and the large language model rule generation model; add the new cleaning rules to the cleaning rule collection to obtain a new rule collection; and update the large language model based on the new rule collection.

[0140] In some embodiments, the processing unit 101 evaluates the first data based on at least one of the following: the degree of grammatical errors corresponding to the first data; the degree of spelling errors corresponding to the first data; the semantic accuracy corresponding to the first data; the completeness of the information corresponding to the first data; the text fluency corresponding to the first data; the vocabulary richness corresponding to the first data; and the accuracy of the emotional color corresponding to the first data.

[0141] In some embodiments, the processing unit 101 is further used to: determine whether the usage time of the large language model reaches a usage time threshold; and / or determine whether the amount of cleaned data of the large language model reaches a quantity threshold.

[0142] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0143] Figure 7 FIG. 1 is a block diagram of a data cleaning device based on a large language model according to an exemplary embodiment, and the device can be represented as an electronic device. Figure 7 As shown, one embodiment of the present disclosure provides a device 200 based on a large language model. The device 200 based on a large language model includes a memory 201, a processor 202, and an input / output (I / O) interface 203. The memory 201 is used to store instructions. The processor 202 is used to call the instructions stored in the memory 201 to execute the data cleaning method of the embodiment of the present disclosure. The processor 202 is connected to the memory 201 and the I / O interface 203, respectively, for example, through a bus system and / or other forms of connection mechanisms (not shown). The memory 201 can be used to store programs and data, including the program of the data cleaning method involved in the embodiment of the present disclosure. The processor 202 executes various functional applications and data processing of the device 200 based on the large language model by running the program stored in the memory 201.

[0144] In the embodiment of the present disclosure, the processor 202 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 202 can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.

[0145] The memory 201 in the embodiment of the present disclosure may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD), etc.

[0146] In the embodiment of the present disclosure, the I / O interface 203 may be used to receive input instructions (such as digital or character information, and key signal input related to user settings and function control of the device 200 based on the large language model, etc.), and may also output various information to the outside (such as images or sounds, etc.). In the embodiment of the present disclosure, the I / O interface 203 may include one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel, etc.

[0147] In some embodiments, the present disclosure provides a storage medium storing computer executable instructions, which, when executed by a processor, perform any of the methods described above.

[0148] In some embodiments, the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, any of the methods described above is performed.

[0149] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0150] The methods and apparatus of the present disclosure can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "apparatus" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0151] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

[0152] The foregoing description of the implementation of the present disclosure has been given for the purpose of illustration and description. The foregoing description is not exhaustive nor is it intended to limit the present disclosure to the exact form disclosed, and various variations and modifications may exist in accordance with the above teachings or may be obtained from the practice of the present disclosure. These embodiments are selected and described in order to illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can utilize the present disclosure in various embodiments and various modifications suitable for the specific purpose contemplated.

[0153] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0154] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0155] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0156] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0157] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0158] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following scope of rights.

[0159] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A data cleaning method based on a large language model, It is characterized in that include: In response to cleaning the data to be cleaned for the i-th time based on the large language model, fine-tuning the large language model used for cleaning the data for the i-th time based on the first rule data matching pair; wherein the first rule data matching pair is determined based on the j-th cleaned data, the data to be cleaned corresponding to the j-th cleaned data, and the cleaning rule collection corresponding to the j-th cleaning; the cleaning rule collection corresponding to the j-th cleaning includes the j-th new cleaning rule, and the j-th new cleaning rule is a cleaning rule determined based on the j-th cleaned data and the data to be cleaned corresponding to the j-th cleaned data; i is a positive integer greater than or equal to 1, j is a positive integer greater than or equal to 1, and i is greater than or equal to j; Generating a cleaning rule matching the current data to be cleaned based on the fine-tuned large language model, and performing data cleaning on the current data to be cleaned to obtain first data; evaluating the first data; In response to an evaluation result of the first data satisfying a requirement, determining that the first data is cleaned data; Generate new cleaning rules based on the data to be cleaned, the first data whose evaluation results meet the requirements, and the large language model rule generation model; Adding the new cleaning rule to the cleaning rule collection to obtain a new rule collection; Based on the new rule collection, the large language model is updated.

2. The method according to claim 1, It is characterized in that The large language model is determined in the following manner: Based on the cleaning rule collection, the manually annotated cleaned data, and the original data to be cleaned corresponding to the manually annotated cleaned data, the prediction large language model is trained to obtain an initial large language model; Cleaning the original data to be cleaned based on the initial large language model and the collection of cleaning rules to obtain second data; Based on the second data, the collection of cleaning rules and the original data to be cleaned corresponding to the second data, a rule data matching pair is obtained, and based on the rule data matching pair, the initial large language model is fine-tuned to obtain a target large language model.

3. The method according to claim 1, It is characterized in that The updating of the large language model based on the new rule collection includes: Generate a new rule-data matching pair based on the data to be cleaned, the new rule collection, and the first data whose evaluation result meets the requirements; The large language model is updated based on the new rule-data matching pair.

4. The method according to claim 1, It is characterized in that The first data is evaluated based on at least one of the following: The degree of grammatical error corresponding to the first data; The degree of spelling errors corresponding to the first data; semantic accuracy corresponding to the first data; The completeness of the information corresponding to the first data; The text fluency corresponding to the first data; The richness of the vocabulary corresponding to the first data; The accuracy of the emotional color corresponding to the first data.

5. The method according to claim 1, It is characterized in that Before fine-tuning the large language model used for the i-th cleaning data based on the first rule data matching pair, the method further includes: Determining that the usage time of the large language model reaches a usage time threshold; and / or It is determined that the amount of cleaned data of the large language model reaches a quantity threshold.

6. A data cleaning device based on a large language model, It is characterized in that include: a processing unit, configured to, when the data to be cleaned is cleaned for the i-th time based on the large language model, fine-tune the large language model used for the i-th cleaning data based on the first rule data matching pair, wherein the first rule data matching pair is determined based on the j-th cleaned data, the data to be cleaned corresponding to the j-th cleaned data, and the cleaning rule collection corresponding to the j-th cleaning, the cleaning rule collection corresponding to the j-th cleaning includes the j-th new cleaning rule, the j-th new cleaning rule is a cleaning rule determined based on the j-th cleaned data and the data to be cleaned corresponding to the j-th cleaned data, the i is a positive integer greater than or equal to 1, the j is a positive integer greater than or equal to 1, and the i is greater than or equal to the j; The data cleaning unit is used to generate cleaning rules that match the current data to be cleaned based on the fine-tuned large language model, and perform data cleaning on the current data to be cleaned to obtain first data; and evaluate the first data, and determine that the first data is cleaned data when the evaluation result of the first data meets the requirements; and is also used to generate new cleaning rules based on the data to be cleaned, the first data whose evaluation result meets the requirements, and the large language model rule generation model; add the new cleaning rules to the cleaning rule collection to obtain a new rule collection; and update the large language model based on the new rule collection.

7. The device according to claim 6, It is characterized in that The large language model is determined in the following manner: based on a cleaning rule collection, manually annotated cleaned data, and original data to be cleaned corresponding to the manually annotated cleaned data, a predicted large language model is trained to obtain an initial large language model; based on the initial large language model and the cleaning rule collection, the original data to be cleaned is cleaned to obtain second data; based on the second data, the cleaning rule collection, and the original data to be cleaned corresponding to the second data, a rule data matching pair is obtained, and based on the rule data matching pair, the initial large language model is fine-tuned to obtain a target large language model.

8. The device according to claim 7, It is characterized in that The data cleaning unit updates the large language model based on the new rule collection in the following manner: Generate a new rule-data matching pair based on the data to be cleaned, the new rule collection, and the first data whose evaluation result meets the requirements; The large language model is updated based on the new rule-data matching pair.

9. The device according to claim 6, It is characterized in that The processing unit evaluates the first data based on at least one of the following: The degree of grammatical error corresponding to the first data; The degree of spelling errors corresponding to the first data; semantic accuracy corresponding to the first data; The completeness of the information corresponding to the first data; The text fluency corresponding to the first data; The richness of the vocabulary corresponding to the first data; The accuracy of the emotional color corresponding to the first data.

10. The device according to claim 7, It is characterized in that The processing unit is also used for: Determining that the usage time of the large language model reaches a usage time threshold; and / or It is determined that the amount of cleaned data of the large language model reaches a quantity threshold.

11. A data cleaning device based on a large language model, It is characterized in that include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the data cleaning method based on the large language model as described in any one of claims 1 to 5.

12. A storage medium, It is characterized in that The storage medium stores instructions. When the instructions in the storage medium are executed by the processor of the terminal, the terminal is enabled to execute the data cleaning method based on the large language model as described in any one of claims 1 to 5.

13. A computer program product comprising a computer program or instructions, It is characterized in that When the computer program or instruction is executed by a processor, the data cleaning method based on a large language model described in any one of claims 1 to 5 is implemented.