A data cleaning method, device, medium and product
Patent Information
- Application Number
- CN202611165133.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-03
- Publication Date
- 2026-10-27
Smart Images
Figure CN122986270A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data cleaning technology, and in particular to a data cleaning method, equipment, medium and product. Background Technology
[0002] With the rapid development of machine learning, advanced artificial intelligence technologies, especially large models, are being increasingly applied to daily life, bringing new development opportunities and changes to various industries. However, these technologies mostly rely on large amounts of structured and unstructured data for training, fine-tuning, and updating. Existing research further shows that the quality of the data used for training has a significant impact on training results and model performance. Therefore, reliable automated data cleaning techniques are becoming increasingly important.
[0003] For some specialized large-scale models, structured data is typically used more frequently than unstructured data. This is mainly because the data collection frequency, output format, and processing flow are often relatively fixed. Early cleaning of structured data focused primarily on identification, i.e., using deep learning algorithms to detect dirty data, followed by manual modification or direct deletion of the corresponding data records. The application of data cleaning techniques has driven the development and implementation of large-scale models, and the development and application of large-scale models, in turn, has promoted the advancement of data cleaning techniques. In recent years, data cleaning methods based on large-scale models have emerged continuously. These methods typically use a small amount of clean data to fine-tune the large model, leveraging the model's understanding of specific content to output the reliability of lexical terms, and further identifying dirty data by setting a reliability threshold. In recent research, researchers have further attempted to use the fine-tuned large model to directly generate the most reliable values to replace dirty data, thereby achieving an integrated approach to identification and correction in structured data cleaning.
[0004] However, existing methods still have some significant drawbacks. First, most existing large-model-based methods cannot cover all cases of unclean data, often only identifying one or a few simpler situations. Second, these methods still have considerable room for improvement in the accuracy of identification and correction, meaning that cleaned data sometimes still requires secondary cleaning using other methods or manually. Finally, the efficiency of existing large-model-based data cleaning methods also needs further improvement. Summary of the Invention
[0005] The purpose of this application is to provide a data cleaning method, equipment, medium, and product that can improve the efficiency of dirty data identification and reduce the workload of human data cleaning, while improving the reliability of corrected data and ensuring the accuracy of correction.
[0006] To achieve the above objectives, this application provides the following solution.
[0007] Firstly, this application provides a data cleaning method, including: Obtain the original dataset to be cleaned, and perform standardized structural representation processing on the original dataset to be cleaned to obtain the serialized sample set to be cleaned; the serialized sample set to be cleaned includes several serialized text samples; Several serialized text samples were selected and manually corrected to obtain manually corrected samples; all manually corrected samples constituted a clean reference sample set. A clean reference sample set is used to train the causal language model, resulting in a structured data cleaning model. A reference index is built based on a clean reference sample set; Input the serialized sample set to be cleaned into the structured data cleaning model, calculate the local negative log-likelihood of each target field in each serialized text sample, and identify suspected dirty fields; By using a structured data cleaning model and a reference index, each suspected dirty field is corrected based on fused retrieval evidence and local context, resulting in a corrected structured dataset.
[0008] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the steps of the data cleaning method described above.
[0009] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data cleaning method described above.
[0010] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data cleaning method described above.
[0011] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a data cleaning method, device, medium, and product. It trains a causal language model using a clean reference sample set, fine-tuning the model to obtain a structured data cleaning model. This structured data cleaning model is then used to clean the serialized sample set to be cleaned, improving the efficiency of dirty data identification. A reference index is constructed based on the clean reference sample set, which helps the structured data cleaning model retrieve the group containing the target field more quickly, better assisting in the identification of suspected dirty fields. By inputting the serialized sample set to be cleaned into the structured data cleaning model, the local negative log-likelihood of each target field in each serialized text sample is calculated to identify suspected dirty fields. This method overcomes the limitations of traditional single-threshold judgment, significantly reducing the probability of false positives and false negatives, improving the efficiency of dirty data identification and reducing the workload of human data cleaning. After identifying suspected dirty fields, the structured data cleaning model and the reference index are used to correct each suspected dirty field based on fused retrieval evidence and local context. This approach can find correction evidence from existing general knowledge and case datasets, and fully combine the context of the data entry containing the dirty data, jointly improving the reliability of the corrected data and ensuring the accuracy of the correction. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a diagram illustrating the application environment of a data cleaning method according to an embodiment of this application. Figure 2 A schematic flowchart illustrating a data cleaning method provided in an embodiment of this application; Figure 3 A distribution diagram of data for correcting errors or omissions provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0014] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0015] The data cleaning method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can send the original dataset to be cleaned to server 104. After receiving the original dataset, server 104 performs standardized structural representation processing on the original dataset to obtain a serialized sample set to be cleaned. Several serialized text samples from the serialized sample set are manually corrected to obtain a clean reference sample set. The clean reference sample set is used to train a causal language model to obtain a structured data cleaning model. Based on the structured data cleaning model, the local negative log-likelihood of each target field in each serialized text sample is calculated to identify suspected dirty fields. The structured data cleaning model and reference index are used to correct each suspected dirty field based on fused retrieval evidence and local context to obtain a corrected structured dataset. Server 104 can then feed back the corrected structured dataset to terminal 102. In addition, in some embodiments, the data cleaning method can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform data cleaning on the original dataset to be cleaned, or the server 104 can obtain the original dataset to be cleaned from the data storage system and perform data cleaning on the original dataset to be cleaned.
[0016] The terminal 102 can be, but is not limited to, various desktop computers and laptops. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers, or it can be a cloud server.
[0017] In one exemplary embodiment, such as Figure 2 As shown, a data cleaning method is provided. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 206.
[0018] Step 201: Obtain the original dataset to be cleaned, and perform standardized structural representation processing on the original dataset to be cleaned to obtain the serialized sample set to be cleaned; the serialized sample set to be cleaned includes several serialized text samples.
[0019] Step 202: Select several serialized text samples for manual correction to obtain manually corrected samples; all manually corrected samples constitute a clean reference sample set.
[0020] Step 203: Train the causal language model using a clean reference sample set to obtain a structured data cleaning model.
[0021] Step 204: Construct a reference index based on a clean reference sample set.
[0022] Step 205: Input the serialized sample set to be cleaned into the structured data cleaning model, calculate the local negative log-likelihood of each target field in each serialized text sample, and identify suspected dirty fields.
[0023] Step 206: Use the structured data cleaning model and reference index to correct each suspected dirty field based on fused retrieval evidence and local context to obtain the corrected structured dataset.
[0024] By implementing steps 201 to 206 above, a causal language model is trained using a clean reference sample set, thereby fine-tuning the model to obtain a structured data cleaning model. This structured data cleaning model is then used to clean the serialized sample set to be cleaned, improving the efficiency of dirty data identification. A reference index is constructed based on the clean reference sample set, which helps the structured data cleaning model retrieve the group containing the target field more quickly, better assisting in the identification of suspected dirty fields. By inputting the serialized sample set to be cleaned into the structured data cleaning model, the local negative log-likelihood of each target field in each serialized text sample is calculated to identify suspected dirty fields. This eliminates the limitations of traditional single-threshold judgment, significantly reducing the probability of false positives and false negatives, improving the efficiency of dirty data identification and reducing the workload of human data cleaning. After identifying suspected dirty fields, each suspected dirty field is corrected based on fused retrieval evidence and local context through the structured data cleaning model and the reference index. This approach can find correction evidence from existing general knowledge and case datasets, and fully combine the context of the data entry containing the dirty data, jointly improving the reliability of the corrected data and ensuring the accuracy of the correction.
[0025] In step 201 above, the original dataset to be cleaned is... Perform standardized structural representation processing to obtain the serialized sample set to be cleaned. and policy documents The serialized sample set to be cleaned The policy file uses a line-by-line JSON format. The policy file is in JSON format. At least the table schema, number of fields, number of data rows, source file identifier, and grouping field identifier should be recorded; the serialized sample set to be cleaned. For text serialization results oriented towards structured representation, it preserves table markers, pattern markers, row markers, field markers, field names, and field values in character-level or string-level form, and uses them to express the structural relationships and field semantics in the original structured data; the serialization sample set to be cleaned. Each serialized text sample in the process remains a directly readable text string at the end of this step, excluding the integer lexical identifier sequence generated by the causal language model segmenter based on the vocabulary mapping, and also excluding the input lexical identifier sequence (input_ids) directly used for model training; based on the policy file Manual verification is performed, and after confirming that there are no pre-set basic structural errors, the serialized sample set to be cleaned is... As input for subsequent steps.
[0026] The specific process of standardizing the structure representation is as follows: 1) When the original dataset to be cleaned Or used to form a clean reference sample set When the original data is not in tabular form, the adaptation module first performs parsing and mapping processing on the corresponding original data. The adaptation module is established based on preset parsing rules and field mapping rules, which are determined according to the format definition, field definition or interface protocol of the original data, so as to parse the original data into a unified internal table structure.
[0027] 2) Perform TOON serialization on the adapted table structure data to ensure that each sample contains at least table tags, schema tags, and row tags, and that each field is represented according to the correspondence between field index, field name, and field value. ;in, Indicates the field index, and , Indicates the total number of fields; Represents a field; Indicates the field name; Represents the value; Indicates the field value.
[0028] The table data is serialized into a JSONL sample, where each row of the table is a complete JSON object, and the text fields of the JSONL sample retain at least the [TABLE] (table), [SCHEMA] (schema), and [ROW] (row) tags; simultaneously, in the policy file... List of record table schemas and total number of fields Total number of rows of data The source file path or identifier, group field name, and text field identifier are provided for manual inspection of infrastructure errors.
[0029] In this step, the jsonl sample output is a structured text intermediate representation. Its function is to unify the expression form of structured data from different sources and preserve the structural semantic boundaries. In this step, the jsonl sample does not perform model word segmentation, does not generate integer word tokens, and does not participate in parameter update calculation. Instead, it is further converted into the model input representation by the word segmenter corresponding to the target causal language model in the subsequent step 2033.
[0030] From the serialized sample set to be cleaned A clean reference sample set is obtained by selecting a manually corrected subset of samples and performing manual correction (correcting erroneous data in the manually corrected subset manually). Wherein, let the serialized sample set to be cleaned be... The total number of samples is The number of samples in the clean reference sample set is ,but Clean reference sample set Retain the row number, field index, field name, and grouping field value corresponding to the original sample for subsequent model fine-tuning and evidence retrieval construction. Training with correct data (a clean reference sample set obtained after manual correction) is essential to obtain the correct probability distribution, thus enabling accurate identification when dealing with dirty data.
[0031] In step 203 above, a clean reference sample set is used to train the causal language model to obtain a structured data cleaning model, specifically including the following steps 2031 to 2034.
[0032] Step 2031: Divide the clean reference sample set into a training subset, a validation subset, and a test subset.
[0033] Step 2032: The serialized text sample is used as the input for word segmentation of the causal language model. The word segmenter corresponding to the causal language model is called. Based on the word segmenter's vocabulary, word segmentation rules and special tag encoding rules, the manually corrected samples of each subset are mapped to word tag sequences.
[0034] Download the initial parameters of the causal language model, the model configuration file, and the word segmenter vocabulary file corresponding to the causal language model from the pre-trained model repository.
[0035] The serialized text sample is used as the input to the model for word segmentation. The word segmenter corresponding to the causal language model is called, and the word segmenter is used based on its vocabulary. The word segmentation rules and special tag encoding rules map each manually corrected sample in the training subset, validation subset, and test subset from a text string to a word tag sequence acceptable to the model, resulting in a word tag sequence. ;in, Indicates the first The integer identifier corresponding to each word element , This indicates the word length of the manually corrected sample, and , The word list represents the word segmenter; the word identifier sequence is generated by the word segmenter after performing word segmentation and word list lookup on the text string output in step 201, and is the result of the first conversion of the manually corrected sample from text representation to model numerical input representation.
[0036] Step 2033: Concatenate multiple lexical identifier sequences of the same subset in a preset order to obtain a long lexical sequence, and then concatenate them according to a preset block length. The long word sequence is divided into fixed-length blocks to obtain the input block sequence of the causal language model; the preset block length satisfies preset constraints ( And it does not exceed the maximum context length of a causal language model; if the length is insufficient... The tail segment is discarded; the sequence of each input block is... The sequence is copied as a supervisory label, enabling the causal language model to predict the next lexical unit at the current position based on the left-hand context during training.
[0037] Step 2034: Fine-tuning training of the causal language model is performed using a full-parameter update method, updating all trainable parameters of the causal language model through backpropagation; after each training round, the validation loss is calculated based on the validation subset, and the model parameters with the minimum validation loss are selected to determine the fine-tuned structured data cleaning model. ,in, This represents the model parameters after fine-tuning.
[0038] Based on a clean reference sample set Build a reference index ; among which, reference index At least include: grouping field values Hehanghao orderly pair An exact row index built for the key, used to return a mapping of field values to the corresponding reference row; grouping field values A grouped row-order index built for the key, used to return a reference row sequence sorted by row number within the same group; and a grouping field value. and field name orderly pair A frequency index of field values built for the key, used to return the mapping relationship between field values and their occurrence counts; reference index. It serves as the common input for precise retrieval, local contextual retrieval, and frequency retrieval in step 206.
[0039] In step 205 above, the serialized sample set to be cleaned is input into the structured data cleaning model, and the local negative log-likelihood of each target field in each serialized text sample is calculated to identify suspected dirty fields. Specifically, this includes the following steps 2051 to 2055.
[0040] Step 2051, denote the structured data cleaning model as... The word sequence corresponding to the serialized text sample is denoted as ,in, Indicates the first Each word element, , This indicates the total length of the lexical sequence.
[0041] Step 2052, for the first The first of the serialized text samples There are 1 target fields, and their term ranges are set to 1. ,in, Indicates the first The starting position of each target field corresponds to the term range of the field. Indicates the first Each target field corresponds to the termination position of a field term interval; the field term interval at least covers the field tag, field index, field name, and field value corresponding to the target field.
[0042] Step 2053, Define the location Prefix cumulative negative log-likelihood for: (1); Among them, the prefix cumulative negative log-likelihood at the initial position ; Representing a structured data cleaning model Given the left-hand context Output words at time The conditional probability. After inputting the serialized sample set to be cleaned into the structured data cleaning model, the structured data cleaning model predicts and outputs the next word at the current position (given the left-hand context) and its conditional probability.
[0043] Step 2054, calculate the first step based on the cumulative negative log-likelihood of the prefix. The first of the serialized text samples Field-specific negative log-likelihood of each target field (unit-specific negative log-likelihood): (2); in, Indicates the first The first of the serialized text samples Local negative log-likelihood of each target field; Indicates the first The cumulative negative log-likelihood of the prefix at the terminating position of the term interval corresponding to each target field; Indicates the first The cumulative negative log-likelihood of the prefix of the position preceding the starting position of the target field's term interval; Step 2055, when the first The first of the serialized text samples When the local negative log-likelihood of the target field is greater than or equal to the suspected dirty field detection threshold, the first target field is determined to be... The first of the serialized text samples The target field is a suspected dirty field. Let the detection threshold for suspected dirty fields be... ,and ,when At that time, the first The first of the serialized text samples The target field is identified as a suspected dirty field and added to the suspected dirty field set. Among them, the set of suspected dirty fields. Each element in the table contains at least the sample row number, field index, field name, original field value, and corresponding field local negative log-likelihood value.
[0044] Targeting the collection of suspected dirty fields For each suspected dirty field, a structured data cleaning model is invoked. An enhanced correction mechanism, integrating retrieval evidence and local context, is executed using precise row indexes, group row order indexes, and field value frequency indexes. Candidate correction values are generated, scored, and a final correction value is determined. This final correction value is then filled back into the corresponding target field, and the corrected structured dataset is output. and cleaning result files Among them, the cleaning result file Record at least the sample row number, field index, field name, original field value, final corrected value, correction strategy, and corresponding comprehensive correction score.
[0045] In step 206 above, the structured data cleaning model and reference index are used to correct each suspected dirty field based on fusion retrieval evidence and local context to obtain the corrected structured dataset, specifically including the following steps 2061 to 2068.
[0046] Step 2061: For each suspected dirty field, retrieve the grouping field value from the reference index based on the serialized text sample to which the suspected dirty field belongs. Extract the set of reference records in the same group.
[0047] Step 2062: Perform row offset alignment using comparable fields and calculate the alignment score for each candidate row offset. Comparable fields refer to the fields selected from the set of fields defined in the table schema of the dataset during row offset alignment, used to compare the corresponding field values of the serialized text sample to be cleaned with those of the clean reference sample.
[0048] Let the first The line order position of the serialized text sample to be cleaned within its respective group is: Its grouping field value is The candidate row offset set is The set of comparable fields obtained from sampling is Then any candidate row offset The alignment score is: (3); in, offset for candidate rows Alignment score; Indicates the first In the nth serialized text sample to be cleaned, the first The field values of each field; The value of the grouping field in the clean reference sample set is... The row order position within the group is The reference sample in the first The field values in each field; Represents the characteristic function.
[0049] Step 2063: Determine the candidate row offset corresponding to the alignment score with the largest value as the optimal row offset, and determine the target reference row based on the optimal row offset.
[0050] Find the optimal row offset ,in, Indicates the first The optimal line offset corresponding to each serialized text sample. Let the line position of the aligned target reference sample within its group be . .
[0051] Step 2064: Based on the target reference row and retrieval evidence, generate four types of candidate value sets, including rule candidate sets generated by field transformation rules, precise retrieval candidate sets based on precise row index matching, local context candidate sets based on same group row order index matching, and frequency candidate sets obtained from high-frequency field values of the same group and field returned by field value frequency index; uniformly perform null value filtering, duplicate value merging, and length constraint filtering on the four types of candidate value sets to obtain the final total candidate set; the final total candidate set includes several candidate values; the retrieval evidence is determined by field transformation rules and reference index.
[0052] 1) Rule candidate set generated by field transformation rules The field transformation rules apply the candidate transformation under the rule constraints to the original field value based on the character composition pattern, value range constraints, format consistency constraints, and coupling relationship between related fields in the same row of the target field, so as to generate candidate values that are consistent with the semantics of the current field and the record context.
[0053] 2) The row position returned by the exact row index, relative to the row order within the group. The precise retrieval candidate set obtained by precisely corresponding the reference field values of the target reference sample ,in, Indicates the first The grouping field values of a serialized text sample to be cleaned; Indicating a clean reference sample set, the grouping field value is... The row order position within the group is The first in the reference sample The field values of each field.
[0054] 3) The row index returned by the same group, located at the th row... Local context candidate set consisting of adjacent reference rows within the row neighborhood ,in, The size of the neighborhood window; For a clean reference sample set, the grouping field value is... The group located before and after the target reference sample Among the adjacent reference samples of the row position within the group, the first... The field values of each field.
[0055] 4) Frequency candidate set obtained from the high-frequency field values of the same group and field returned by the field value frequency index. ,in, This indicates that in the clean reference sample set, the grouping field value is... In the reference sample, the field field values The number of times a value appears; the frequency candidate set retains at most the top 12 high-frequency field values; This indicates that the frequencies are sorted from largest to smallest, and the first one is selected. The grouping field value is In the reference sample, the field The field value.
[0056] Then, null value filtering, duplicate value merging, and length constraint filtering are performed on each of the above candidate value sets to obtain the final total candidate set. .
[0057] Step 2065: Calculate the evidence score and unit word probability cost for each candidate value, and determine the comprehensive corrected score for each candidate value.
[0058] Let the first The first of the serialized text samples Each target field is used to replace candidate values. The resulting candidate field word sequence is ,in, Indicates candidate value The first in the corresponding field word sequence Each word element, Indicates candidate value The length of the corresponding field word sequence.
[0059] Let the left-hand prefix context preceding the target field be denoted as ,in, For the first The starting position of the term interval corresponding to each target field.
[0060] Candidate values are calculated based on a structured data cleaning model. In prefix context Approximate conditional probability (4); in, Indicates candidate value In prefix context The approximate conditional probability is given below.
[0061] Calculate candidate values based on approximate conditional probability. The unit word probability cost is calculated using the following formula: (5); in, Candidate values The unit word probability cost; This represents the candidate values calculated by the structured data cleaning model. In prefix context Approximate conditional probability under the following conditions; prefix context For the first The first of the serialized text samples The left-hand prefix context preceding each target field; Indicates candidate value The length of the corresponding field word sequence.
[0062] The formula for calculating the overall corrected score of candidate values is as follows: (6); (7); (8); in, Candidate values The overall corrected score; For weight parameters, ; Candidate values Evidence score; , , and Indicates the weight coefficients of various types of search evidence; It is an indicator function; Indicates belonging to; For the rule candidate set; To accurately retrieve the candidate set; For local context candidate set; This represents the number of candidate values in the local context candidate set. This represents a positive integer to prevent the denominator from being zero. Indicates candidate value Compared with the current candidate values to be scored same; express Any candidate value; Indicating a clean reference sample set, the grouping field value is... In the reference sample, the first Each field has a value The number of times it appears; Indicating a clean reference sample set, the grouping field value is... In the reference sample, the first Each field can take the candidate value. The number of times it appears; This indicates that the grouping field value in the clean reference sample set is... In the reference sample, the first The maximum number of occurrences corresponding to the field values of each field, where For fields any field value; Candidate values The amount of improvement, Indicates candidate value The probability cost is better than the original field value; Original field value The unit word probability cost; Candidate values The unit word probability cost.
[0063] Step 2066: Determine the initial candidate values based on the comprehensive corrected scores of all candidate values. (Following...) The candidate value with the highest comprehensive corrected score is selected as the initial candidate value. .
[0064] Step 2067: Determine the final correction value for the suspected dirty field based on the unit token probability cost of the candidate values in the precise retrieval candidate set and the unit token probability cost of the initial candidate values. This applies when precise retrieval candidate values exist (candidate values in the precise retrieval candidate set). And satisfy At that time, the final corrected value is determined as the candidate value in the exact search candidate set, that is... Otherwise, the final corrected value will be determined as the initial candidate value, i.e. Among them, the tolerance for precise retrieval of candidate values. satisfy .
[0065] Step 2068: Replace the suspected dirty fields in the serialized text sample with the final corrected value to perform correction, and obtain the corrected serialized text sample; all the corrected serialized text samples constitute the corrected structured dataset.
[0066] The effectiveness of the above method is verified below: The original dataset to be cleaned in step 201 can be the hospital_100k dataset, which is an open-source medical dataset (containing both dirty data and clean data corrected manually). The dataset is already an Excel spreadsheet, so no adapter is needed in this case. The field_name includes: prno is the hospital number, hospitalname is the hospital name, address, city, state, and zip code represent the hospital address, city, state, and zip code, respectively. countryname is the county / county name. phone is the hospital phone number, hospitaltype is the hospital type, hospitalowner is the hospital owner or operator, and emergencyservice indicates whether emergency services are provided. condition is the category to which the medical quality indicator belongs, such as surgical infection prevention; mc is the specific indicator code; measurename is the text description of the indicator; score is the hospital's score on the indicator; sample is the case sample size used to calculate the indicator; stateavg is a combination of the state code and the specific indicator code. Each field_name contains errors (dirty data).
[0067] The causal language model in step 203 above can be GPT-2 Small, which has very few parameters and can ensure efficiency. In addition, this application conducts experiments based on a model with a very small number of parameters. The proportions of the training subset, validation subset and test subset are 80%, 15% and 5%, respectively.
[0068] Repeated experiments were conducted on the hospital_100k dataset to determine the threshold for detecting suspected dirty fields. Take 2.5, neighborhood window size Take 2, The tolerance for exact search candidate values is 12. It is 0.15; The value is 512.
[0069] The above content is implemented in Python and involves Python libraries, including third-party libraries such as Pillow, absl, brotli, datasets, einops, imageio, matplotlib, natsort, numpy, pandas, peft, psutil, pydub, torch, torchvision, tqdm, and transformers.
[0070] After fine-tuning GPT-2 Small, the dirty data version (serialized text samples) corresponding to the original validation and training subsets (clean version) was selected as input data. In this input, a total of 31,545 dirty data objects were identified, of which 2,257 correct data were incorrectly identified as dirty data, and 311 errors were missed. The algorithm's accuracy was 91.85%, recall was 98.95%, and F1 score was 95.80%. Of the 31,545 identified dirty data, 29,288 were corrected, resulting in a 100% correction accuracy. Compared to the existing best algorithm for this dataset, Cocoon, GPT-2 maintains its lead in all metrics, as shown in Table 1.
[0071] Table 1. Comparison of metrics between the method of this application and the Cocoon algorithm.
[0072] The distribution of corrected or missing data is as follows Figure 3 As shown, this is a pie chart with 311 missing data points. Among them, the number of missing hospital addresses is 242, accounting for 79.1%; the number of missing hospital numbers is 42, accounting for 13.7%; the number of missing hospital phone numbers is 16, accounting for 5.2%; the number of missing hospital cities is 3, accounting for 1.0%; and the number of missing case sample sizes is 3, accounting for 1.0%.
[0073] This application has achieved excellent results with a very small number of model parameters, verifying that the proposed method has higher accuracy and reliability. Therefore, it can be expected to achieve even better performance with a large number of parameters, further demonstrating the feasibility and excellent performance of the algorithm.
[0074] This application identifies dirty data based on a fine-tuned causal data cleaning model (causal LM) and NLL index, and introduces an enhanced correction mechanism that integrates retrieval evidence and local context to improve the dirty data correction performance of the fine-tuned causal LM.
[0075] This application has the following advantages: (1) The structured data cleaning model with fine-tuning is used to output the probability distribution of word units, and NLL is further calculated. The comparison between NLL and the set threshold is used to automatically determine whether the data is dirty data (suspected dirty field), thereby improving the recognition efficiency and reducing the human data cleaning workload. (2) An enhanced correction mechanism that integrates retrieval evidence and local context is introduced. This mechanism can find correction evidence from existing general knowledge and case datasets, and can also fully combine the context of the data bar where the dirty data is located to improve the reliability of the correction data and ensure the accuracy of the correction. (3) Provides a correction logic for multiple candidate values to ensure that the most appropriate correction value is used and to avoid the use of the second-best candidate correction value.
[0076] The data cleaning method provided in this application can be applied to fields such as healthcare, financial risk control, government and enterprise services, industrial manufacturing, scientific research, e-commerce retail (product information maintenance), legal compliance (extraction of key information from documents), and human resources (employee data integration).
[0077] This application also provides an application scenario in which the above-mentioned data cleaning method is applied. Specifically, the data cleaning method provided in this embodiment can be applied to a medical and health data cleaning scenario. The medical and health data cleaning scenario includes a content production stage, a content processing chain, and a content distribution stage. The original dataset to be cleaned enters the content processing chain from the content production stage, and after human-machine collaboration, a corrected structured dataset is obtained, which then enters the downstream content distribution stage. The data cleaning method provided in this embodiment belongs to the content processing chain. Specifically, in the content processing chain for the original dataset to be cleaned, the original dataset to be cleaned can be standardized into a structural representation to obtain a serialized sample set to be cleaned. Several serialized text samples from the serialized sample set to be cleaned are manually corrected to obtain a clean reference sample set. The clean reference sample set is used to train a causal language model to obtain a structured data cleaning model. Based on the structured data cleaning model, the local negative log-likelihood of each target field in each serialized text sample is calculated to identify suspected dirty fields. The structured data cleaning model and reference index are used to correct each suspected dirty field based on fused retrieval evidence and local context to obtain a corrected structured dataset.
[0078] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores the data to be cleaned and the corrected structured dataset. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data cleaning method.
[0079] Figure 4 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0080] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0081] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.
[0083] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0084] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0086] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A data cleaning method, characterized in that, The data cleaning method includes: Obtain the original dataset to be cleaned, and perform standardized structural representation processing on the original dataset to be cleaned to obtain the serialized sample set to be cleaned; the serialized sample set to be cleaned includes several serialized text samples; Several serialized text samples were selected and manually corrected to obtain manually corrected samples; all manually corrected samples constituted a clean reference sample set. A clean reference sample set is used to train the causal language model, resulting in a structured data cleaning model. A reference index is built based on a clean reference sample set; Input the serialized sample set to be cleaned into the structured data cleaning model, calculate the local negative log-likelihood of each target field in each serialized text sample, and identify suspected dirty fields; By using a structured data cleaning model and a reference index, each suspected dirty field is corrected based on fused retrieval evidence and local context, resulting in a corrected structured dataset.
2. The data cleaning method according to claim 1, characterized in that, The causal language model is trained using a clean reference sample set to obtain a structured data cleaning model, which specifically includes: The clean reference sample set is divided into a training subset, a validation subset, and a test subset; The serialized text samples are used as input to the causal language model for word segmentation. The corresponding word segmenter of the causal language model is called. Based on the word segmenter's vocabulary, word segmentation rules and special tag encoding rules, the manually corrected samples of each subset are mapped to word tag sequences. Multiple lexical identifier sequences from the same subset are concatenated in a preset order to obtain a long lexical sequence. The long lexical sequence is then divided into blocks of a fixed length according to a preset block length to obtain the input block sequence of the causal language model. The preset block length satisfies a preset constraint and does not exceed the maximum context length of the causal language model. The input_ids of each input block sequence are copied as a supervision label sequence, enabling the causal language model to predict the next word at the current position based on the left-hand context; The causal language model is fine-tuned using a full parameter update method, and all trainable parameters of the causal language model are updated via backpropagation. After each round of training, the validation loss is calculated based on the validation subset, and the model parameters with the minimum validation loss are selected to determine the structured data cleaning model.
3. The data cleaning method according to claim 1, characterized in that, Reference indexes include exact row indexes built with ordered pairs of grouping field values and row numbers as keys, same-group row order indexes built with grouping field values as keys, and field value frequency indexes built with ordered pairs of grouping field values and field names as keys.
4. The data cleaning method according to claim 1, characterized in that, The serialized sample set to be cleaned is input into the structured data cleaning model. For each target field in each serialized text sample, the local negative log-likelihood of that field is calculated to identify potentially dirty fields. Specifically, this includes: Let the structured data cleaning model be denoted as... The word sequence corresponding to the serialized text sample is denoted as ,in, Indicates the first Each word element, , Indicates the total length of the lexical sequence; For the The first of the serialized text samples There are 1 target fields, and their term ranges are set to 1. ,in, Indicates the first The starting position of each target field corresponds to the term range of the field. Indicates the first Each target field corresponds to the termination position of a field term interval; the field term interval at least covers the field tag, field index, field name, and field value corresponding to the target field; Define position Prefix cumulative negative log-likelihood for: ； in, Representing a structured data cleaning model Given the left-hand context Output words at time The conditional probability; Calculate the first based on the prefix cumulative negative log-likelihood. The first of the serialized text samples Local negative log-likelihood of each target field: ； in, Indicates the first The first of the serialized text samples Local negative log-likelihood of each target field; Indicates the first The cumulative negative log-likelihood of the prefix at the terminating position of the term interval corresponding to each target field; Indicates the first The cumulative negative log-likelihood of the prefix of the position preceding the starting position of the target field's term interval; When the The first of the serialized text samples When the local negative log-likelihood of the target field is greater than or equal to the suspected dirty field detection threshold, the first target field is determined to be... The first of the serialized text samples The target field is suspected to be a dirty field.
5. The data cleaning method according to claim 3, characterized in that, Using a structured data cleaning model and reference index, each suspected dirty field is corrected based on fused retrieval evidence and local context, resulting in a corrected structured dataset, specifically including: For each suspected dirty field, extract the set of reference records in the same group from the reference index based on the grouping field value of the serialized text sample to which the suspected dirty field belongs; Row offset alignment is performed using comparable fields, and alignment scores are calculated for each candidate row offset. The candidate row offset corresponding to the alignment score with the largest value is determined as the optimal row offset, and the target reference row is determined based on the optimal row offset; Based on the target reference row and retrieval evidence, four types of candidate value sets are generated: a rule candidate set generated by field transformation rules, a precise retrieval candidate set based on precise row index matching, a local context candidate set based on same group row order index matching, and a frequency candidate set obtained from high-frequency field values of the same group and field returned by the field value frequency index. Null value filtering, duplicate value merging, and length constraint filtering are uniformly performed on the four types of candidate value sets to obtain the final total candidate set. The final total candidate set includes several candidate values. The retrieval evidence is determined by the field transformation rules and the reference index. For each candidate value, calculate the evidence score and unit word probability cost, and determine the comprehensive corrected score for each candidate value. Initial candidate values are determined based on the combined corrected scores of all candidate values. The final correction value of the suspected dirty field is determined based on the unit word probability cost of the candidate values in the precise retrieval candidate set and the unit word probability cost of the initial candidate values; The final corrected value replaces the suspected dirty fields in the serialized text sample, and the corrected serialized text sample is obtained; all the corrected serialized text samples constitute the corrected structured dataset.
6. The data cleaning method according to claim 5, characterized in that, The formula for calculating the unit word probability cost of candidate values is as follows: ； in, Candidate values The unit word probability cost; This represents the candidate values calculated by the structured data cleaning model. In prefix context Approximate conditional probability under the following conditions; prefix context For the first The first of the serialized text samples The left-hand prefix context preceding each target field; Indicates candidate value The length of the corresponding field word sequence.
7. The data cleaning method according to claim 5, characterized in that, The formula for calculating the overall corrected score of candidate values is as follows: ； ； ； in, Candidate values The overall corrected score; These are weight parameters; Candidate values Evidence score; , , and Indicates the weight coefficients of various types of search evidence; It is an indicator function; Indicates belonging to; For the rule candidate set; To accurately retrieve the candidate set; For local context candidate set; This represents the number of candidate values in the local context candidate set. This represents a positive integer to prevent the denominator from being zero. Indicates candidate value Compared with the current candidate values to be scored same; express Any candidate value; Indicating a clean reference sample set, the grouping field value is... In the reference sample, the first Each field has a value The number of times it appears; Indicating a clean reference sample set, the grouping field value is... In the reference sample, the first Each field can take the candidate value. The number of times it appears; This indicates that the grouping field value in the clean reference sample set is... In the reference sample, the first The maximum number of occurrences corresponding to the field values of each field, where For fields any field value; Candidate values The amount of improvement; Original field value The unit word probability cost; Candidate values The unit word probability cost.
8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the data cleaning method according to any one of claims 1-7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data cleaning method according to any one of claims 1-7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data cleaning method according to any one of claims 1-7.